Code Project: Make talking RSS feeds

Code

On the face of it, writing a script/program to download and parse an RSS feed, and from there send news items to a speech synthesizer, sounds ambitious - even for TuxRadar. But as it turns out, it's actually rather straightforward.

Principally, this is thanks to three technologies, Python, Festival and Linux. Python, the world-dominating scripting language par-excellence makes it easy to construct a script without too much thought or effort. The open source Festival Speech Synthesis System sounds fantastic, and can be installed with just a couple of clicks from your distribution's package manager. And Linux itself; without its powerful pipes and process scheduling, we'd have to spend a lot more time writing that functionality into our program, and we'd also need to add a GUI to make it all easily accessible.

Luckily, all we need to do is write a small Python script and use a little command line magic to tie all these things together. We're going to write a simple script in Python that will output plain text news stories than can then be piped into Festival, which will then speak the news through your speaker or headphones. This gives you maximum flexibility. This two-pronged approach (Python script piped to Festival) can be modified to suit almost any purpose. In less than an hour, you'll be able to sit back and listen to the dulcet tones of a female voice synthesizer reading the latest happenings from TuxRadar.com.

Part One: Talking heads

The first thing we need to do is get the speech synthesizer working. This might feel like we're doing the last step first, but it's only after the speech synthesizer is installed that we'll be able to build and test our program. Festival packages are included in most popular Linux distributions, and this is just as well if you take a look at the convoluted requirements and naming conventions used on the project's website.

Festival can output a variety of languages and dialects, from Hindi and Marathi to Czech and Italian, which means the package you need to install will depend on your locale. For UK-English, for example, you'll need to grab both the 'Festival lexicon from the Oxford Advanced Learners' Dictionary' and the 'Part of speech lexicons and ngram from English' packages, as well as the vanilla Festival package and a 'speaker' (voice) package of your choice. Depending on the popularity of your language, you can expect to find male and female voices, at low (8k) and high (16k) sample rates.

The range of Festival packages is bewildering, but you don't need them all for a working installation.

The range of Festival packages is bewildering, but you don't need them all for a working installation.

If you've never used Festival before, this is going to feel a little arcane. Getting the software to speak seems to have been a secondary motive to the developers. Type festival on the command line, and you'll be presented with an interpreter interface. To say something, type (SayText "Hello World!"), including the brackets.

If everything is installed and running correctly, you should hear the gargling tones of the Festival speech synthesizer mouthing out the words "hello" and "world". This voice is entirely dependent on the packages you installed. If you want to scare yourself by browsing the amount of options Festival offers, just type help.

We need Festival to speak to us from the command line, and there are two different methods. The first uses the --tts argument. This command will either read a file (literally!) or read from the terminal's standard input. This means we can pipe the output of any command into Festival.

For example, typing this:

echo "Hello world" | festival --tts

...will create the same output as before, piping the output from the echo command into Festival without launching the interpreter, which will then send the sound to your speakers. But, there's a problem with this tactic. If the soundcard is already being used, you'll get an error: "Linux: can't open /dev/dsp", because Festival doesn't interface with the Linux sound system, it simply send the raw audio data to /dev/dsp. If that's being used, it won't work. And most desktop environments like to take over /dev/dsp so they can playback that 'ping' sound when you get a new message.

The solution is a little convoluted. First, we launch Festival in server mode - type "festival --server". This forces the application to run in the background, silently waiting for requests to convert text to speech from a Festival client. This is more efficient than starting Festival each time you want to say something, and it also opens up remote talking possibilities.

To connect to the server and generation speech, we need to use the festival_client command in another terminal, along with the --ttw argument. Rather than sending the audio to the sound device directly (and failing if the device is already in use), the ttw option will send the raw audio data to the console. We need to pipe this to a command that can read this output and send it to the audio layer.

The easiest option is a command called aplay, which you should find installed by default. Aplay talks to the ALSA layer directly, which means you can share your audio device with other applications. The complete command using the client and the server, and sending the output to aplay is echo "Hello World" | festival_client --ttw | aplay. When you type this on the command line, you should hear the output from the Festival speech synthesizer. We're now ready to tackle the programming.

Part 2: RSS decoding

With sound generation covered, we can focus on programming; starting with downloading and parsing an RSS feed. This is the part that initially sounds daunting, but it's actually very easy to implement thanks to something called 'Feedparser' - a Python module that can download and parse all the most common RSS and Atom feed formats.

The KDE text editor, Kate, does a good job of highlighting Python syntax and marking loops and functions with the border brackets.

The KDE text editor, Kate, does a good job of highlighting Python syntax and marking loops and functions with the border brackets.

As it's a module, you'll need to install it separately, but because of its universal brilliance, you'll find packages bundled along with Python in your distribution's package manager. If you've not used Python before, you're going to find it a refreshing change to the world of dependencies, libraries, headers and inheritance that blights other more 'corporate' programming languages.

For example, to import Feedparser functionality into our own program, we simply need to start our program with import feedparser. This is the only line we need to be able to start accessing RSS feeds from our Python program. To prove this, we'll add a couple of more lines to grab the title of the first feed from our own website. Open your favourite text editor and add the following to a text file beneath the above import command:

import feedparser
rss_url = "http://www.tuxradar.com/rss"
feed = feedparser.parse( rss_url )
print feed.entries[0].title

Save this text to a file ending with the py extension, and run it from the command line by typing python filename. You should see the title from the top news story from the TuxRadar website printed to the console. This is all thanks to Feedparser. It grabs the feed from tuxradar.com, and returns the data as a Python Unicode string, which we've assigned to the feed variable. It's from this that all other RSS/Atom data is derived.

In the above example, we output the title of the first news story. Each story is an item in the entries array, and we're free to query almost any element from the feed. If we replaced title with description, we'd get the body of the news story rather than just the title.

There are many elements like these we can use, depending on the version and format of the feed. Try adding print feed to output the entire contents of the RSS source you're using, and look through the output for elements that might be useful. Common elements include feed.title for the title of the feed, feed.link to return the URL of the hosting website, feed.date to return the date of the feed.

To make the process more streamlined, Python scripts can be made to run automatically; place this on the first line of the source code

#!/usr/bin/python 

and making the file executable with

chmod +x filename.py 

on the command line. Your script can now be run by typing ./filename.py on the command line, and we can integrate this into our Festival configuration by typing:

./filename.py | festival_client --ttw | aplay

You should now be hearing the title of our top news story being read to you by the Festival speech synthesizer, and that's the hardest part to the whole process. From this point, you could adapt our simple script to pick and choose the articles you want to hear, traverse through a collection of feeds, and any number of other enhancements. But before we do that, we need to create the framework in our script to help this happen. Let's start again with a blank file and create a more flexible solution.

RSS vs Atom

We've got used to the idea that the small orange icon we see for syndicated news on a website represents an RSS feed. But there are many competing versions and formats that hide behind the same symbol. The two most common are RSS 2.0 and Atom, and both are fully supported by the Feedparser module used in our script.

Atom was developed to remedy several perceived problems with RSS. In RSS, there's no way to state which language is used in a single RSS story, for example, or whether the story is published as HTML or plain text. But this disparity has created problems for developers.

The two different formats need to be treated slightly differently. This is because Atom contains elements that RSS does not, and even the formatting of common elements can be markedly different - such as with date and times. You need to make sure the feed you're parsing supports the elements you're expecting in the format you need. And the only way you can be sure of this is to download the raw feed data.

You can do this by either downloading the file pointed to by the feed's URL, or using a news aggregator to grab the raw data and save it to disk. You'll then be able to take a look at the feed's contents in a regular text editor.

Part 3: Python programming

Top of the usability hit list is the ability to parse command line options. If we add this to our program, we'll be able to use different arguments to grab alternative feeds, specify the number of top stories to read as well as operating modes.

With these options implemented, we'll then be able to automate our program through cron, or through other scripts using the internal arguments to change the function of the script - just as we do with other utilities.

Python's argument parsing (as it's called), is very similar to that of the C and C++ languages, and uses a module called getopt. This needs to be added to the import instruction at the top of our script, along with another module called sys and Feedparse from before.

Sys provides basic system-specific functionality, and is a required addition for all but the most basic scripts. We now need to split our script into functions, so that we can call each individually, as well as deal with the command line arguments. As with C/C++, we'll start with the main function, which is where the main flow of program logic lives.

def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], "hu:c:t", ["help", "url=", "count=", "title"])
    except getopt.GetoptError, err:
        print str(err)
        usage()
        sys.exit(2)

The first line of the main function declares its existence, def main():, before parsing the command line arguments between the peculiar try and except statements. These offer a basic form of exception handling. The try statement parses the command line arguments, looking for the single letter arguments h, u: c: or t, and their longer counterparts help, url=, count= and title.

We'll use these arguments to add functionality to the script. Following the script with either -h or --help, for instance, will print usage information for the script. url="http://feedurl" will tell the script the URL of the feed to grab, while -c 3 or --count=3 will tell the script to only grab the latest three stories. Finally, the -t and --title arguments act as a switch. When enabled, the script will only grab the title of each news story rather than the title and the description.

If any arguments aren't recognised, the exception segment is run, printing a standard error, before running the usage() function and exiting. Now we've detected and parsed the command line arguments used to execute the script, we need to do something useful with them. Using a for statement, we'll skip through the command line arguments and assign their values to variables we can use in the RSS-grabbing part of the script. Beneath the exception handler, add the following chunk of code:

    url = "http://www.tuxradar.com/rss"
    title = False
    count = 1
	
    for o, a in opts:
        if o in ("-h", "--help"):
            noargument()
            sys.exit()
        elif o in ("-u", "--url"):
            url = a
        elif o in ("-c", "--count"):
            count = int(a)
        elif o in ("-t", "--title"):
            title = True
        else:
            assert False, "invalid option"

We create default values for the three variables we're going to use url for location of the RSS feed, title to decide whether to include the description of each news story or just the title, and count for the number of stories to read. The if statement goes through each of the command line options and arguments (o and a in the code).

If -h or --help is detected, the noargument function is run and the script exits. If -u or --url= is detected, the value of the URL (the argument) as passed to the url variable. If either -c- or --count= is detected, then count = int(a) converts the argument string to an integer and assigns that value to the count variable. Finally, if -t or --title is detected, the title flag is set to true.

Making sense of these configuration options is very easy with the Feedparser code we used in our initial script. We just use another for loop to grab each story according to the value of the count variable (which defaults to 1), and decide whether we need to grab the story description as well as the title. Here's the code to do it, and it should follow the previous chunk:

    feed = feedparser.parse( url )
    feed['feed']['title']
    for i in range(0, count):
        if title:
            print feed['entries'][i]['title']
        else:
            print feed['entries'][i]['title']
            print feed['entries'][i]['description']

All we're doing in this section is grabbing each news story according to the value of count, and outputting the text to the standard output. This will be the terminal, and it's from there we can pipe the text into the Festival speech synthesizer. That's all we need to include in the main function. All that's left to add is a little housekeeping. Beneath the above piece of code, add the following ( a required addition for any Python script with a main() function:

if __name__ == "__main__":
    sys.exit(main())

If you've been keeping on top of the other functions we've been calling, you'll have noticed we used two functions we haven't written yet - usage() when the exception handler doesn't detect a valid argument, and noargument when the user asks for --help. Both of these functions need to give the user a little nudge in the right direction, firstly by suggesting the user adds --help and secondly be documenting each basic function of the script. To implement both functions, add something similar to the following at the top of your source code file, between the import statement, and the beginning of the def main(): function.

def usage():
    print "Try 'rss2voice --help' for more information."

def noargument():
    print "Download text from an RSS feed and send it to the standard output."
    print "-h, --help prints this information"
    print "-u, --url=the URL for the RSS feed"
    print "-c --count= number of stories you need"
    print "-t,	--title output only the title of each story"

And that's all there is to the code. You've now written a fully functional RSS/Atom parsing script you can used to send text to the Festival speech synthesizer. As usual in such a small amount of space, we've omitted most error checking - the worst offender is that we don't check the validity of any URL that is passed to the script.

This will create problems if there's no RSS feed, and may generate errors within the script. Otherwise, everything is ready to go. Just replace the new script with the old one in the original command we used to talk through Festival, and you'll hear all the latest news stories from TuxRadar spoken to you in mellifluous tones.

If you're looking for more ideas for things to try, here are some suggestions:

  • Create a crontab addition to run the script automatically
  • Add a wait mode to look for new stories every hour
  • Mark articles as read or unread to avoid duplication
  • Configure a different speech synthesizer or voice
  • Integrate the Festival speech directly into the script
The problem with scripts that only create sound is that there's nothing interesting to look at in a screenshot - you'll just have to imagine the drunken tones of the Festival speech synthesizer telling us about Windows vs Ubuntu.

The problem with scripts that only create sound is that there's nothing interesting to look at in a screenshot - you'll just have to imagine the drunken tones of the Festival speech synthesizer telling us about Windows vs Ubuntu.

First published in Linux Format

First published in Linux Format magazine

You should follow us on Identi.ca or Twitter


Your comments

command tags...

I wish you put the commands (such as echo "Hello World" | festival_client --ttw | aplay) mentioned inside the paragraphs in proper (like <code> or <pre> or <tt>) text styles...

I just thought of this

I recently suggested this very feature to Akregator!!
OMG! I love you TuxRadar!!

problem

I tried to use this program but when i use the
import feedparser
it gives me the following error

import feedparser
ImportError: No module named feedparser

can anyone help me

installing feedparser

just a note as I found out - when you get the above "no module named feedparser: error; go and install feedparser.

feedparser.org (download it here)

then, unpack the zip.

go to command line

type:

python setup.py install

it all worked quite well for me until I wanted it to speak, then I got this:

monkey@primateserver:~$ python ./feedparserdd.py | festival_client --ttw | aplay
socket: connect failed
aplay: playback:2208: read error
close failed in file object destructor:
Error in sys.excepthook:

Original exception was:

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar