IPython Magic Functions

Both Python for Data Science for Dummies and Machine Learning for Dummies rely on a version of Anaconda that uses IPython as part of its offering.Theoretically, you could also use Anaconda with Beginning Programming with Python For Dummies, but that book is designed to provide you with an experience that relies on the strict Python offerings (without the use of external tools). In other words, the procedures in this third book are designed for use with IDLE, the IDE that comes with Python. IPython extends the development environment in a number of ways, one of which is the use of magic functions. You see the magic functions in the code of the first two books as calls that begin with either one or two percent signs (% or %%). The most common of these magic functions is %matplotlib, which controls how IPython Notebook or Jupyter Notebook display plot output from the code.

You can find a listing of the most common magic functions in the Python for Data Science for Dummies Cheat Sheet. Neither of the first two books use any other magic functions, so this is also a complete list of magic functions that you can expect to find in our books. However, you might want to know more. Fortunately, the site at https://damontallen.github.io/IPython-quick-ref-sheets/ provides you with a complete listing of the magic commands (and a wealth of other information about IPython).

Of course, you might choose to use another IDE—one that isn’t quite so magical as Anaconda provides through IPython. In this case, you need to remove those magic commands. Removing the commands won’t affect functionality of the code. The example will still work as explained in the book. However, the way that the IDE presents output could change. For example, instead of being inline, plots could appear in a separate window. Even though using a separate window is less convenient, either method works just fine. If you ever do encounter a magic function-related problem, please be sure to let me know at John@JohnMuellerBooks.com.

 

Spaces in Paths

A number of readers have recently written me about an error they see when attempting to compile or execute an application or script in books such as, C++ All-In-One for Dummies, 3rd EditionBeginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies. Development environments often handle spaces differently because they’re designed to perform tasks such as compiling applications and running scripts. I had touched on this issue once before in the Source Code Placement post. When you see an error message that tells you that a file or path isn’t found, you need to start looking at the path and determine whether it contains any spaces. The best option is to create a directory to hold your source code and to place that directory off the root directory of your drive if at all possible. Keeping the path small and simple is your best way to avoid potential problems compiling code or running scripts.

The problem for many readers is that the error message is buried inside a whole bunch of nonsensical looking text. The output from your compiler or interpreter can contain all sorts of useful debugging information, such as a complete listing of calls that the compiler, interpreter, or application made. However, unless you know how to read this information, which is often arcane at best, it looks like gobbledygook. Simply keep scanning through the output until you see something that humans can read and understand. More often than not, you see an error message that helps you understand what went wrong, such as not being able to find a file or path. Please let me know if you ever have problems making the code examples in my books work, but also be sure to save yourself some time and effort by reading those error messages. Let me know if you have any thoughts or concerns about spaces in directory paths at John@JohnMuellerBooks.com.

 

Installing Python Packages (Part 2)

In the Installing Python Packages (Part 1) post, you discovered the easiest method of installing new packages when working with Beginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies. Using the pip command is both fast and easy. However, it doesn’t provide much in the way of feedback when things go wrong. To overcome this issue, you can use the conda command in place of pip when you have Anaconda installed on your system. Like pip, conda supports a wide variety of commands. You can find a listing of these commands at http://conda.pydata.org/docs/using/pkgs.html.

You need to know a few things about working with conda. The first is that you need to open an Anaconda prompt to use it. For example, when working with Windows, you use the Start ⇒ All Programs ⇒ Anaconda<Version> ⇒ Anaconda Prompt command to open a window like the one shown here where you can enter commands. (Your Anaconda Prompt may look different than the one shown based on the platform you use and the version of Anaconda you have installed.)

Use the Anaconda Prompt to gain access to the conda command.
The Anaconda Prompt

You can easily discover the features the conda command supports by typing conda -h and pressing Enter. You see a list of command line switches similar to the ones shown here:

Use the conda command line switches to perform various tasks.
A Listing of Conda Switches

As you can see, there are quite a few tasks you can perform. To determine whether you have a package installed, use the Conda search <package name> command.  For example, if you want to determine if you have Pandas installed, you type Conda search Pandas and press Enter.  You see a list of Pandas versions installed, assuming that Pandas is installed, like this:

Use the search switch to locate a particular package installation.
A Listing of Pandas Information

The information you get from conda is far more in depth than pip provides. To determine what you have installed, just go down the list and determine whether you have the version of Pandas that you need.  If you don’t, then type Conda update pandas and press Enter (notice the case used).  On the other hand, let’s say you want to install BeautifulSoup.  Well, the first time through, try typing Conda install BeautifulSoup and pressing Enter.  You see an error message that tells you what to type like this:

The conda command provides you with helpful error information.
Using Error Information

Since you want to install the latest BeautifulSoup, type Conda install beautiful-soup and press Enter.  After searching for the required update information, conda will ask if you want to proceed.  Type y and press Enter.  You’ll see a whole bunch of activity take place, but eventually, you have a new version of BeautifulSoup, plus all the supporting functionality, installed correctly in the correct locations.  Here’s how things looked on my system:

Conda provides detailed information about the installation process.
Viewing the Result of an Installation

At this point, you have BeautifulSoup installed. Installing other packages follows the same path. Using conda does require a little more expertise than using pip, but you also gain additional flexibility and garner more information. When everything goes well, either tool does an equally good job of getting the installation or update task done, but conda excels in helping you past troublesome installations. Let me know your thoughts about using conda to install the packages required by my books at John@JohnMuellerBooks.com.

 

Installing Python Packages (Part 1)

My Python-related books, Beginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies use various libraries to perform book-specific tasks. The books do provide instructions as needed, but, based on reader input, sometimes these instructions aren’t as clear as necessary, located in precisely the right location, or possibly as specific as needed. This post will help you get the packages containing the libraries you need installed in order to get more from the books.

It’s essential to remember that Beginning Programming with Python for Dummies relies on the 3.3.4 version of Python. The other two books rely on Python 2.7.x versions. The reason for using the older version of Python in these two books is that these books rely on libraries that Python 3.x doesn’t support. If you try to install these libraries on Python 3.x, you’ll get an error message of somewhat dubious usefulness.

In most cases, the easiest way to install a package is to open a command prompt with Administrator privileges and rely on the pip (for Python 2.x) or pip3 (for Python 3.x) command to perform the installation. For example, to install BeautifulSoup, you can type pip install beautifulsoup4 and press Enter. Installing any other package follows about the same route.

The only problem with the pip utility is that you don’t get it with every version of Python. When using an older version of Python, such as 3.3.4, you actually need to install the pip utility to use it. Fortunately, the installation instructions at https://pip.pypa.io/en/latest/installing/ aren’t difficult to use and you’ll be up and running in a few minutes.

Some readers have also complained that pip doesn’t provide much information when it comes to errors. The lack of information can prove problematic when an installation doesn’t go as planned. Next week I plan to cover the conda utility that comes with Anaconda. This utility isn’t as easy to use in some respects as pip, but it does provide considerably more information. If you have any questions about using the pip utility with my books, please contact me at John@JohnMuellerBooks.com.

 

Mac Gatekeeper Error

A number of my books, such as C++ All-In-One for Dummies, 3rd EditionBeginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies ask readers to download an IDE or other code and install it on their Mac systems. The problem is that the Mac system won’t always cooperate. For example, you might see an error dialog like the one shown for Code::Blocks:

The Gatekeeper error tells you that it won't allow you to install software from unknown publishers.
Your Mac won’t let you install software.

The problem is one of permissions. The default permissions set for newer Mac systems restrict you to getting your apps from the Mac App Store or from vendors who have signed their files. Fortunately, you can overcome this problem either temporarily or permanently, depending on how you want to use your Mac. The Fix the “App can’t be opened because it is from an unidentified developer” Error in Mac OS X blog post provides you with illustrated, step-by-step directions to perform the task using either method. Let me know if you encounter any other problems getting your Mac to install the software required to use my books at John@JohnMuellerBooks.com.

 

Working with Code in e-Books

Most of my technical readers now use e-books instead of paper books. Of course, there is a convenience factor to storing your entire library on a Kindle, even if it’s a software version of the Kindle. Of course, there are all sorts of e-book formats for your desktop system as well. The point is that electronic format makes a lot of sense when dealing with technical books.

However, e-books can cause some interesting problems and I’ve encountered a few with a number of readers now. The most important consideration is that you can’t cut and paste code from an e-book directly into your IDE and expect it to work. There are all sorts of reasons for this exclusion. For example, cutting and pasting may insert special characters into the output stream or the resulting paste may not have white space in the right places. A common problem is that publishers often convert regular single and double quotes into curly quote equivalents. The two kinds of quotes (both single and double) are completely different and the second type definitely won’t compile.

The best option when working with an e-book is to view the code in the e-book, but still get the downloadable source code for the book from the publishers website. I always provide a blog post detailing where to obtain the downloadable source for a book, when you need source code to use the book. If you can’t find the downloadable source, always feel free to contact me at John@JohnMuellerBooks.com. I want to be sure you have a great reading experience, which means having source code that actually runs in your development environment.

Another potential problem with e-books is that you may see unfortunate code breaks (despite the efforts of the publisher and myself). When you need to understand how white space works with a programming language, always review the downloadable source. The fact that the downloadable source compiles and runs tells you that all the of white space is in the right place and of the correct type. Typing the source code directly out of your e-book could result in added carriage returns or other white space errors that will cause the code to fail, even though the commands, variables, and other parts of the code are all correct.

As always, I’m open to your questions about my books. If you don’t understand how things work, please contact me—that’s why I’m here.

 

Is Security Research Always Useful?

Anyone involved in the computer industry likely spends some amount of time reading about the latest security issues in books such as Security for Web Developers. Administrators and developers probably spend more time than many people, but no one can possibly read all the security research available today. There are so many researchers looking for so many bugs in so many places and in so many different ways that even if someone had the time and inclination to read every security article produced, it would be impossible. You’d need to be the speediest reader on the planet (and then some) to even think about scratching the surface. So, you must contemplate the usefulness of all that research—whether it’s actually useful or simply a method for some people to get their name on a piece of paper.

Some of the attacks require physical access to the system. In some cases, you must actually take the system apart to access components in order to perform the security trick. Unless you or your organization is in the habit of allowing perfect strangers physical access to your systems, which might include taking them apart, you must wonder whether the security issue is even worth worrying about. You need to ask why someone would take the time to document a security issue that’s nearly impossible to see, much less perform in a real world environment. More importantly, the moment you see that a security issue requires physical access to the device, you can probably stop reading.

You also find attacks that require special equipment to perform. The article, How encryption keys could be stolen by your lunch, discusses one such attack. In fact, the article contains a picture of the special equipment that you must build to perpetrate the attack. It places said equipment into a piece of pita bread, which adds a fanciful twist to something that is already quite odd and pretty much unworkable given that you must be within 50 cm (19.6 in) from the device you want to attack (assuming that the RF transmission conditions are perfect). Except for the interesting attack vector (using a piece of pita bread), you really have to question why anyone would ever perpetrate this attack given that social engineering and a wealth of other attacks require no special equipment, are highly successful, and work from a much longer distance.

Another example of incredibly weird security research is found in the article, When the good guys are wielding the lasers. I have to admit it’s interesting in a James Bond sort of way, but we’re talking about lasers mounted on drones. This attack at least has the advantage of distance (1 km or 0.6 mi). However, you have to wonder just how the laser was able to get a line of sight with the attack object, a printer in this case. If a device is critical enough that someone separates it from the Internet, it’s also quite likely that the device won’t be sitting in front of a window where someone can use a laser to access it.

A few research pieces become more reasonable by discussing outlandish sorts of hacks that could potentially happen after an initial break-in. The hack discussed in Design flaw in Intel chips opens door to rootkits is one of these sorts of hacks. You can’t perpetrate the hack until after breaking into the system some other way, but the break-in has serious consequences once it occurs. Even so, most hackers won’t take the time because they already have everything needed—the hack is overkill.

The articles that help most provide a shot of reality into the decidedly conspiracy-oriented world of security. For example, Evil conspiracy? Nope, everyday cyber insecurity, discusses a series of events that everyone initially thought pointed to a major cyber attack. It turns out that the events occurred at the same time by coincidence. The article author thoughtfully points out some of the reasons that the conspiracy theories seemed a bit out of place at the outset anyway.

It also helps to know the true sources of potential security issues. For example, the articles, In the security world, the good guys aren’t always good and 5 reasons why newer hires are the company’s biggest data security risk, point out the sources you really do need to consider when creating a security plan. These are the sorts of articles that should attract your attention because they describe a security issue that you really should think about. Likewise, reading articles such as, Software developers aren’t implementing encryption correctly and 4 fatal problems with PKI help you understand why your security measures may not always work as well as anticipated.

The point is that you encounter a lot of information out there that doesn’t help you make your system any more secure. It may be interesting if you have the time to read it, but the tactics truly aren’t practical and no hacker is going to use them. Critical thinking skills are your best asset when building your security knowledge. Let me know about your take on security research at John@JohnMuellerBooks.com.

 

Warnings in Python and Anaconda

It seems as if Python developers are having more than a few problems at the moment from a number of sources. I recently wrote about the potential issues for readers of Beginning Programming with Python For Dummies and Python for Data Science for Dummies from Windows 10 (Python and Windows 10). However, some readers have come back afterward to say they’re still seeing warnings. It wasn’t until one of the beta readers for Machine Learning for Dummies also saw some of these warnings that it became apparent that some other problem is at work. A recent upgrade to NumPy 1.10.1 has created these warnings. You can see some message threads about the issue at:

The important thing to remember is that you’ll see warnings, not errors (unless there is a problem Luca, my coauthor for Python for Data Science for Dummies, and I haven’t seen yet). For now, updating all of the Anaconda components is the only way to actually get rid of the warnings, which can prove to be quite a pain. However, the warnings are just that, warnings. The code in the books will still run just fine. The best way to avoid a lot of work and potentially creating yet more problems is to ignore the warnings for now. In order to ignore the warnings, type the following two lines of code:

import warnings
warnings.simplefilter("ignore")

Obviously, the situation is inconvenient for everyone, but the various libraries will get in sync sometime soon and then the warnings will disappear until the next set of updates. Please let me know if you continue to see problems after making this fix at John@JohnMuellerBooks.com.

 

Python and Windows 10

A number of Beginning Programming with Python For Dummies and Python for Data Science for Dummies readers have written to tell me that the installation instructions for Python in these two books don’t appear to work well with Windows 10. Unfortunately, Windows 10 wasn’t available during the writing of either book, but the operating system does seem to present problems for a number of people—not just developers. Microsoft’s enforced upgrades are just one source of woe. Of course, Windows 10 has its supporters as well who are trying to tell you not to worry about these issues. I’m not here to tell you whether you should use Windows 10 or not—that’s a topic for another post. However, I also understand you need a fix for the installation process for these two books if you are running Windows 10.

For the most part, all you really need to do is install Python 3.x for Beginning Programming with Python for Dummies and Anaconda for Python for Data Science for Dummies. The problem doesn’t appear to be the actual installation (given there are no error messages when the installation completes), but rather accessing the applications after the installation. To ensure you can access the applications, you need to be sure they’re part of the path. You may also need to open a command prompt to start the applications, rather than rely on a Start Menu entry to access them. Given that I don’t have Windows 10 installed and don’t plan to install it for now because I need to support the documented configurations for the books, the best I can do is direct you to a site where you can discover how to perform these tasks under Windows 10. The article I suggest is: Setting up your Windows 10 System for Python Development (PyDev, Eclipse, Python). You don’t need to setup Eclipse or do anything else fancy. Once you have Python installed, you should be ready to go.

My feeling is that Windows 10 is going to create more than a few problems for developers because the forced upgrades will mean that you can’t ever rely on your setup being stable. The moment you get one set of Microsoft induced problems fixed, the operating system will automatically download a new set to your machine. For this reason, I can’t recommend using Windows 10 for development purposes. You’ll be better served with Windows 7 or Windows 8, with Windows 7 being the optimal choice. It could be that I’m wrong on this issue and I do plan to explore it further, but for the moment, I’m not offering Windows 10 support directly. I’ll do what I can to get you up and running with your Windows 10 system, but I can’t guarantee results because my books haven’t been written with the vagaries of Windows 10 in mind. Please let me know about your book-specific questions and concerns at John@JohnMuellerBooks.com.

 

Source Code Placement

I always recommend that you download the source code for my books. The Verifying Your Hand Typed Code post discusses some of the issues that readers encounter when typing code by hand. However, I also understand that many people learn best when they type the code by hand and that’s the point of getting my books—to learn something really interesting (see my principles for creating book source code in the Handling Source Code in Books post). Even if you do need to type the source code in order to learn, having the downloadable source code handy will help you locate errors in your code with greater ease. I won’t usually have time to debug your hand typed code for you.

Depending on your platform, you might encounter a situation the IDE chooses an unfortunate place to put the source code you want to save. For example, on a Windows system it might choose the C:\Program Files folder (or a subdirectory) to the store the file. Microsoft wants to make your computing experience safer, so you don’t actually have rights to this folder for storing your data file. As a result, the IDE will stubbornly refuse the save the files in that folder. Likewise, some IDEs have a problem with folder names that have spaces in them. For example, your C:\Users\<Your Name>\My Documents folder might seem like the perfect place to store your source code files, but the spaces in the path will cause problems for the IDE and it will claim that it can’t find the file, even if it manages to successfully save the file.

My recommendation for fixing these, and other source code placement problems, is to create a folder that you can access and have full rights to work with to store your source code. My books usually make a recommendation for the source code file path, but you can use any path you want. The point is to create a path that’s:

  • Easy to access
  • Allows full rights
  • Lacks spaces in any of the path name elements

As long as you follow these rules, you likely won’t experience problems with your choice of source code location. If you do experience source code placement problems when working with my books, please be sure to let me know at John@JohnMuellerBooks.com.