IPython Magic Functions

Both Python for Data Science for Dummies and Machine Learning for Dummies rely on a version of Anaconda that uses IPython as part of its offering.Theoretically, you could also use Anaconda with Beginning Programming with Python For Dummies, but that book is designed to provide you with an experience that relies on the strict Python offerings (without the use of external tools). In other words, the procedures in this third book are designed for use with IDLE, the IDE that comes with Python. IPython extends the development environment in a number of ways, one of which is the use of magic functions. You see the magic functions in the code of the first two books as calls that begin with either one or two percent signs (% or %%). The most common of these magic functions is %matplotlib, which controls how IPython Notebook or Jupyter Notebook display plot output from the code.

You can find a listing of the most common magic functions in the Python for Data Science for Dummies Cheat Sheet. Neither of the first two books use any other magic functions, so this is also a complete list of magic functions that you can expect to find in our books. However, you might want to know more. Fortunately, the site at https://damontallen.github.io/IPython-quick-ref-sheets/ provides you with a complete listing of the magic commands (and a wealth of other information about IPython).

Of course, you might choose to use another IDE—one that isn’t quite so magical as Anaconda provides through IPython. In this case, you need to remove those magic commands. Removing the commands won’t affect functionality of the code. The example will still work as explained in the book. However, the way that the IDE presents output could change. For example, instead of being inline, plots could appear in a separate window. Even though using a separate window is less convenient, either method works just fine. If you ever do encounter a magic function-related problem, please be sure to let me know at [email protected].


Installing Python Packages (Part 2)

In the Installing Python Packages (Part 1) post, you discovered the easiest method of installing new packages when working with Beginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies. Using the pip command is both fast and easy. However, it doesn’t provide much in the way of feedback when things go wrong. To overcome this issue, you can use the conda command in place of pip when you have Anaconda installed on your system. Like pip, conda supports a wide variety of commands. You can find a listing of these commands at http://conda.pydata.org/docs/using/pkgs.html.

You need to know a few things about working with conda. The first is that you need to open an Anaconda prompt to use it. For example, when working with Windows, you use the Start ⇒ All Programs ⇒ Anaconda<Version> ⇒ Anaconda Prompt command to open a window like the one shown here where you can enter commands. (Your Anaconda Prompt may look different than the one shown based on the platform you use and the version of Anaconda you have installed.)

Use the Anaconda Prompt to gain access to the conda command.
The Anaconda Prompt

You can easily discover the features the conda command supports by typing conda -h and pressing Enter. You see a list of command line switches similar to the ones shown here:

Use the conda command line switches to perform various tasks.
A Listing of Conda Switches

As you can see, there are quite a few tasks you can perform. To determine whether you have a package installed, use the Conda search <package name> command.  For example, if you want to determine if you have Pandas installed, you type Conda search Pandas and press Enter.  You see a list of Pandas versions installed, assuming that Pandas is installed, like this:

Use the search switch to locate a particular package installation.
A Listing of Pandas Information

The information you get from conda is far more in depth than pip provides. To determine what you have installed, just go down the list and determine whether you have the version of Pandas that you need.  If you don’t, then type Conda update pandas and press Enter (notice the case used).  On the other hand, let’s say you want to install BeautifulSoup.  Well, the first time through, try typing Conda install BeautifulSoup and pressing Enter.  You see an error message that tells you what to type like this:

The conda command provides you with helpful error information.
Using Error Information

Since you want to install the latest BeautifulSoup, type Conda install beautiful-soup and press Enter.  After searching for the required update information, conda will ask if you want to proceed.  Type y and press Enter.  You’ll see a whole bunch of activity take place, but eventually, you have a new version of BeautifulSoup, plus all the supporting functionality, installed correctly in the correct locations.  Here’s how things looked on my system:

Conda provides detailed information about the installation process.
Viewing the Result of an Installation

At this point, you have BeautifulSoup installed. Installing other packages follows the same path. Using conda does require a little more expertise than using pip, but you also gain additional flexibility and garner more information. When everything goes well, either tool does an equally good job of getting the installation or update task done, but conda excels in helping you past troublesome installations. Let me know your thoughts about using conda to install the packages required by my books at [email protected].


Installing Python Packages (Part 1)

My Python-related books, Beginning Programming with Python For Dummies, Python for Data Science for Dummies, and Machine Learning for Dummies use various libraries to perform book-specific tasks. The books do provide instructions as needed, but, based on reader input, sometimes these instructions aren’t as clear as necessary, located in precisely the right location, or possibly as specific as needed. This post will help you get the packages containing the libraries you need installed in order to get more from the books.

It’s essential to remember that Beginning Programming with Python for Dummies relies on the 3.3.4 version of Python. The other two books rely on Python 2.7.x versions. The reason for using the older version of Python in these two books is that these books rely on libraries that Python 3.x doesn’t support. If you try to install these libraries on Python 3.x, you’ll get an error message of somewhat dubious usefulness.

In most cases, the easiest way to install a package is to open a command prompt with Administrator privileges and rely on the pip (for Python 2.x) or pip3 (for Python 3.x) command to perform the installation. For example, to install BeautifulSoup, you can type pip install beautifulsoup4 and press Enter. Installing any other package follows about the same route.

The only problem with the pip utility is that you don’t get it with every version of Python. When using an older version of Python, such as 3.3.4, you actually need to install the pip utility to use it. Fortunately, the installation instructions at https://pip.pypa.io/en/latest/installing/ aren’t difficult to use and you’ll be up and running in a few minutes.

Some readers have also complained that pip doesn’t provide much information when it comes to errors. The lack of information can prove problematic when an installation doesn’t go as planned. Next week I plan to cover the conda utility that comes with Anaconda. This utility isn’t as easy to use in some respects as pip, but it does provide considerably more information. If you have any questions about using the pip utility with my books, please contact me at [email protected].


Is Security Research Always Useful?

Anyone involved in the computer industry likely spends some amount of time reading about the latest security issues in books such as Security for Web Developers. Administrators and developers probably spend more time than many people, but no one can possibly read all the security research available today. There are so many researchers looking for so many bugs in so many places and in so many different ways that even if someone had the time and inclination to read every security article produced, it would be impossible. You’d need to be the speediest reader on the planet (and then some) to even think about scratching the surface. So, you must contemplate the usefulness of all that research—whether it’s actually useful or simply a method for some people to get their name on a piece of paper.

Some of the attacks require physical access to the system. In some cases, you must actually take the system apart to access components in order to perform the security trick. Unless you or your organization is in the habit of allowing perfect strangers physical access to your systems, which might include taking them apart, you must wonder whether the security issue is even worth worrying about. You need to ask why someone would take the time to document a security issue that’s nearly impossible to see, much less perform in a real world environment. More importantly, the moment you see that a security issue requires physical access to the device, you can probably stop reading.

You also find attacks that require special equipment to perform. The article, How encryption keys could be stolen by your lunch, discusses one such attack. In fact, the article contains a picture of the special equipment that you must build to perpetrate the attack. It places said equipment into a piece of pita bread, which adds a fanciful twist to something that is already quite odd and pretty much unworkable given that you must be within 50 cm (19.6 in) from the device you want to attack (assuming that the RF transmission conditions are perfect). Except for the interesting attack vector (using a piece of pita bread), you really have to question why anyone would ever perpetrate this attack given that social engineering and a wealth of other attacks require no special equipment, are highly successful, and work from a much longer distance.

Another example of incredibly weird security research is found in the article, When the good guys are wielding the lasers. I have to admit it’s interesting in a James Bond sort of way, but we’re talking about lasers mounted on drones. This attack at least has the advantage of distance (1 km or 0.6 mi). However, you have to wonder just how the laser was able to get a line of sight with the attack object, a printer in this case. If a device is critical enough that someone separates it from the Internet, it’s also quite likely that the device won’t be sitting in front of a window where someone can use a laser to access it.

A few research pieces become more reasonable by discussing outlandish sorts of hacks that could potentially happen after an initial break-in. The hack discussed in Design flaw in Intel chips opens door to rootkits is one of these sorts of hacks. You can’t perpetrate the hack until after breaking into the system some other way, but the break-in has serious consequences once it occurs. Even so, most hackers won’t take the time because they already have everything needed—the hack is overkill.

The articles that help most provide a shot of reality into the decidedly conspiracy-oriented world of security. For example, Evil conspiracy? Nope, everyday cyber insecurity, discusses a series of events that everyone initially thought pointed to a major cyber attack. It turns out that the events occurred at the same time by coincidence. The article author thoughtfully points out some of the reasons that the conspiracy theories seemed a bit out of place at the outset anyway.

It also helps to know the true sources of potential security issues. For example, the articles, In the security world, the good guys aren’t always good and 5 reasons why newer hires are the company’s biggest data security risk, point out the sources you really do need to consider when creating a security plan. These are the sorts of articles that should attract your attention because they describe a security issue that you really should think about. Likewise, reading articles such as, Software developers aren’t implementing encryption correctly and 4 fatal problems with PKI help you understand why your security measures may not always work as well as anticipated.

The point is that you encounter a lot of information out there that doesn’t help you make your system any more secure. It may be interesting if you have the time to read it, but the tactics truly aren’t practical and no hacker is going to use them. Critical thinking skills are your best asset when building your security knowledge. Let me know about your take on security research at [email protected].


Warnings in Python and Anaconda

It seems as if Python developers are having more than a few problems at the moment from a number of sources. I recently wrote about the potential issues for readers of Beginning Programming with Python For Dummies and Python for Data Science for Dummies from Windows 10 (Python and Windows 10). However, some readers have come back afterward to say they’re still seeing warnings. It wasn’t until one of the beta readers for Machine Learning for Dummies also saw some of these warnings that it became apparent that some other problem is at work. A recent upgrade to NumPy 1.10.1 has created these warnings. You can see some message threads about the issue at:

The important thing to remember is that you’ll see warnings, not errors (unless there is a problem Luca, my coauthor for Python for Data Science for Dummies, and I haven’t seen yet). For now, updating all of the Anaconda components is the only way to actually get rid of the warnings, which can prove to be quite a pain. However, the warnings are just that, warnings. The code in the books will still run just fine. The best way to avoid a lot of work and potentially creating yet more problems is to ignore the warnings for now. In order to ignore the warnings, type the following two lines of code:

import warnings

Obviously, the situation is inconvenient for everyone, but the various libraries will get in sync sometime soon and then the warnings will disappear until the next set of updates. Please let me know if you continue to see problems after making this fix at [email protected].


Python and Windows 10

A number of Beginning Programming with Python For Dummies and Python for Data Science for Dummies readers have written to tell me that the installation instructions for Python in these two books don’t appear to work well with Windows 10. Unfortunately, Windows 10 wasn’t available during the writing of either book, but the operating system does seem to present problems for a number of people—not just developers. Microsoft’s enforced upgrades are just one source of woe. Of course, Windows 10 has its supporters as well who are trying to tell you not to worry about these issues. I’m not here to tell you whether you should use Windows 10 or not—that’s a topic for another post. However, I also understand you need a fix for the installation process for these two books if you are running Windows 10.

For the most part, all you really need to do is install Python 3.x for Beginning Programming with Python for Dummies and Anaconda for Python for Data Science for Dummies. The problem doesn’t appear to be the actual installation (given there are no error messages when the installation completes), but rather accessing the applications after the installation. To ensure you can access the applications, you need to be sure they’re part of the path. You may also need to open a command prompt to start the applications, rather than rely on a Start Menu entry to access them. Given that I don’t have Windows 10 installed and don’t plan to install it for now because I need to support the documented configurations for the books, the best I can do is direct you to a site where you can discover how to perform these tasks under Windows 10. The article I suggest is: Setting up your Windows 10 System for Python Development (PyDev, Eclipse, Python). You don’t need to setup Eclipse or do anything else fancy. Once you have Python installed, you should be ready to go.

My feeling is that Windows 10 is going to create more than a few problems for developers because the forced upgrades will mean that you can’t ever rely on your setup being stable. The moment you get one set of Microsoft induced problems fixed, the operating system will automatically download a new set to your machine. For this reason, I can’t recommend using Windows 10 for development purposes. You’ll be better served with Windows 7 or Windows 8, with Windows 7 being the optimal choice. It could be that I’m wrong on this issue and I do plan to explore it further, but for the moment, I’m not offering Windows 10 support directly. I’ll do what I can to get you up and running with your Windows 10 system, but I can’t guarantee results because my books haven’t been written with the vagaries of Windows 10 in mind. Please let me know about your book-specific questions and concerns at [email protected].


Download Site for Python

I recently received an e-mail from a reader who had a bad install with Python 3.3.4 on a laptop with 64-bit Windows 7 installed. No matter what the reader did, the installation wouldn’t work. The application would fail with an error stating that pythonw.exe was unable to start and it included an error of 0xc000007b. He had downloaded the code from https://www.python.org/download/releases/3.3.4/, which is the site mentioned on page 25 of Beginning Programming with Python For Dummies. However, downloading a copy from http://continuum.io/downloads#py34 or https://store.continuum.io/cshop/anaconda/ did provide a copy of Python 3.4.3 (not the version 3.3.4 that is used in the book) that does work on his system.

The problem with this solution is that installing a copy from this second site also installs Anaconda—a product that isn’t covered in the book. In order to work with the IDLE examples in the book, you must open a copy of IDLE in the Anaconda\Scripts folder of the Anaconda installation. You’ll likely find this folder in your personal folder of your system. If you do find that you can’t get the copy of the product from the Python download site to work on your system, try this second solution and please let me know about the issue at [email protected]. I would strongly encourage you to try the setup found in the book, however, because using Anaconda will cause extra work for you and this book is truly meant to help someone who has little or no programming experience discover the joys of working with Python.

As a side note, I have tried the book’s source code with the latest Python release, 3.4.3 (the book was originally written to use version 3.3.4). All of the source code works on my test system, but I’d love to hear if it works on your system as well. You can obtain this updated version of Python at https://www.python.org/downloads/release/python-343/ or http://continuum.io/downloads#py34 (if you don’t mind installing Anaconda as well).

When using the 3.4.3 version of Python, your screenshots may vary some from those found in the book. All version-specific information will change, so you need to take this change into account as you read. Please let me know if you experience any problems using this updated version on your system. In the meantime, happy reading!


Security = Scrutiny

There is a myth among administrators and developers that it’s possible to keep a machine free of viruses, adware, Trojans, and other forms of malware simply by disconnecting it from the Internet. I’m showing my age (yet again), but machines were being infected with all sorts of malware long before the Internet became any sort of connectivity solution for any system. At one time it was floppy disks that were the culprit, but all sorts of other avenues of attack present themselves. To dismiss things like evil USB drives that take over systems, even systems not connected to the Internet, is akin to closing your eyes and hoping an opponent doesn’t choose to hit you while you’re not looking. After all, it wouldn’t be fair. However, whoever said that life was fair or that anyone involved in security plays by the rules? If you want to keep your systems free of malware, then you need to be alert and scrutinize them continually.

Let’s look at this issue another way. If you refused to do anything about the burglar rummaging around on the first floor while you listened in your bedroom on the second floor, the police would think you’re pretty odd. The first thing they’ll ask you is why you don’t have an alarm system implemented into your home. Or if you do have one, wouldn’t it have been a good idea to set it in the first place, so more people would have been notified about this security breach. In addition to alarm systems, some homeowners also have the Best CCTV Camera Kits installed around their homes. They would be able to provide a good image of the burglar. However, it’s still important to try and do something to actually stop the burglar. Whatever you do, you can’t just stand back and do nothing. More importantly, you’d have a really hard time getting any sort of sympathy or empathy from them. After all, if you just let a burglar take your things while you blithely refuse to acknowledge the burglar’s presence, whose fault is that? (Getting bonked on the back of the head while you are looking is another story.) That’s why you need to monitor your systems, even if they aren’t connected to the Internet. Someone wants to ruin your day and they’re not playing around. Hackers are dead serious about grabbing every bit of usable data on your system and using it to make your life truly terrible. Your misery makes them sublimely happy. Really, take my word for it.

The reason I’m discussing this issue is that I’m still seeing stories like, “Chinese hacker group among first to target networks isolated from Internet.” So, what about all those networks that were hacked before the Internet became a connectivity solution? Hackers have been taking networks down for a considerable time period and it doesn’t take an Internet connection to do it. The story is an interesting one because the technique used demonstrates that hackers don’t have to be particularly good at their profession to break into many networks. It’s also alarming because some of the networks targeted were contractors for the US military.

There is no tool, software, connection method, or secret incantation that can protect your system from determined hackers. I’ve said this in every writing about security. Yes, you can use a number of tools to make it more difficult to get through and to dissuade someone who truly isn’t all that determined. Unfortunately, no matter how high you make the walls of your server fortress, the hacker can always go just a bit further to climb them. Headlines such as “Advanced Attackers go Undetected for a Median of 229 Days; Only One-Third of Organizations Identify Breaches on Their Own” tell me that most organizations could do more to scrutinize their networks. Every writing I read about informed security is that you can’t trust anyone or anything when you’re responsible for security, yet organizations continue to ignore that burglar on the first floor.

There is the question of whether it’s possible to detect and handle every threat. The answer is that it isn’t. Truly gifted hackers will blindside you can cause terrifying damage to your systems every time. Monitoring can mitigate the damage and help you recover more quickly, but the fact is that it’s definitely possible to do better. Let me know your thoughts about security at [email protected].

Using Pass Versus Continue in Python

A number of people have asked me about the discussion of the pass and continue clauses on page 140 of Beginning Programming with Python For Dummies. The example on that page is confusing a lot of people. Most people assume that when the example prints its output, the w should not appear as part of the output—as if the pass and continue clauses work precisely the same.

If you look at the second sentence of the first paragraph on page 140, you see that it tells you that pass and continue work almost the same way, except that the pass clause allows completion of the code in the if block in which it appears. This distinction is important. The continue clause is immediate, the pass clause isn’t. So, yes, you can achieve different results using pass or continue.

However, both clauses work in the same way in that they stop execution of the current loop and continue with the next loop. The difference is when they stop execution of the current loop and it all hinges on the if statement block.

Enough people have written about this particular example that I want to be sure there is no confusion about the difference between pass and continue. Please let me know if you have any additional questions about these two clauses at [email protected].



Beginning Python for Dummies Chapter 13 Error

Even with the most carefully crafted book, errors do creep in (see Errors in Writing). There is an error in Chapter 13 of Beginning Programming with Python For Dummies on page 247. In Step 8 you are supposed to, “Type MyTuple[4] and press Enter.” The output information for that step is wrong. Instead of seeing Orange, as specified in the book, you see Yellow. The value Yellow was added to the tuple in Step 7 as the fifth value, which you access using MyTuple[4]. I’m sincerely sorry about any problems that the error may have caused in using the book. Please let me know if you have any questions about this issue at [email protected].