Web Scraping With lxml

Submitted by sudheer on Sun, 08/22/2010 - 22:17

More and more websites are offering APIs nowadays. Previously, we've talked about XML-RPC and REST. Even though web services are growing exponentially there are a lot of websites out there that offer information in unstructured format. Especially, the government websites. If you want to consume information from those websites, web scraping is your only choice.

What is web scraping?

Web scraping is a technique used in programs that mimic a human browsing the website. In order to scrape a website in your programs you need tools to

  • Make HTTP requests to websites
  • Parse the HTTP response and extract content

Make Your Own Script Appender In Mako Templates

Submitted by sudheer on Wed, 07/21/2010 - 19:50

In a recently started Pylons project, I wanted to make an easy script appending facility in Mako templates.

The requirement:

  • base.mako contains the layout of the web page. Many templates inherit base.mako. Here's a snippet from base.mako
    <html>
<head>
    <title>Some title</title>
    <script>...</script>
    <script>...</script>
</head>
 
</%def>
  • my_page.mako inherits base.mako. From within my_page.mako we want to be able to append script tags in the head section of the web page.

Becoming Productive In Bash Using The Keyboard Shortcuts

Submitted by sudheer on Mon, 07/05/2010 - 00:02

Moving around

You can use the arrow keys on keyboard to move around in the command line. Bash also provides convenient keyboard short cuts to navigate effectively. Try them out and see for yourself.

To become a Bash pro user you have to get yourself familiar with the keyboard shortcuts. Once you do, you'll find yourself productive.

CTRL+b move backward one character
CTRL+f move forward one character
ESC+b move one word backward
ESC+f move one word forward
CTRL+a move to beginning of line
CTRL+e move to end of line
CTRL+p move to previous line
CTRL+n move to next line
ESC+< move to first line of history list
ESC+> move to last line of history list

Moving around words using ESC+f and ESC+b are my favourites in this list. Jumping to first and last lines of the history list is also useful.


Programatically Create DateTextBox

Submitted by sudheer on Wed, 06/16/2010 - 15:45

How to create dijit.form.DateTextBox widget programmatically

There are two ways to create Dojo's widgets

  1. Declarative
  2. Programmatic

It is often a matter of project preference and personal opinion to decide which approach to take. Many developers are against mixing up markup and JavaScript.

Programmatically creating widgets has its advantages. For instance, you may want to create a date picker when a button is clicked.

In a previous article we discussed how to add a Dojo date picker declaratively. We also showed how to add Dojo date picker without writing a singe line of JavaScript using the Zend Framework.

Let's create a dijit.form.DateTextBox widget programmatically step by step.


PHP 5 e-commerce Development - Book Review

Submitted by sudheer on Sun, 05/23/2010 - 20:57

I was contacted by PackT to review the book PHP 5 e-commerce Development by Michael Peacock.

The book serves as an introductory tutorial on developing an e-commerce website using PHP. The book has 15 chapters covered in 310 pages.

You can grab a sample chapter from the publisher's website.

The publisher's website has a detailed table of contents.

Who should read the book?

You should read the book if you are learning PHP and new to e-commerce. Beginners trying to utilize out of the box software like Drupal CMS or OSCommerce tend to be frustrated sooner or later. These content management systems have their own ways of doing things. Being new to PHP and complex software like Drupal can intimidate you until you thoroughly understand the inner workings of the software. Often developers choose to roll their own software to avoid the steep learning curve of existing open source software. If you have experienced similar feeling you can sure try this book.


A Bit Of XML, RSS And CURL In 7 Lines Of PHP And A Useful Program

Submitted by sudheer on Wed, 03/31/2010 - 23:34

Today, I was looking for a quick way to get the current weather information on my computer. There are so many websites out there that offer the information. But I was looking for a program I could permanently install on my computer and launch it whenever I want to lookup the weather information. Oddly, I didn't find any satisfying program. At the same time I was also watching a video about network programming. That inspired me to quickly write a program in PHP to print the current weather information where I live.

I started to look out for a web service that offers information about weather for free. Did I tell you programmableweb.com is a useful website to find web services? If you have subscribed to the Tech Chorus blog you know we've been talking about REST, XML-RPC and web services in general for a while. I landed up on the Yahoo! Weather API web page.

I wrote a program to print the weather information in 7 lines of PHP code. I have published this program on Code Album github repository. You can grab it and use it.

If you want to know how to write similar programs, read on. If you know a bit of PHP and have heard about XML and RSS before you can understand the program and start building upon it.


Create RESTful Applications Using The Zend Framework - Part III : Managing API Key

Submitted by sudheer on Mon, 03/01/2010 - 00:52

In the first two posts of this series, we discussed how to route REST requests to controllers and return HTTP response code. In this article I will talk about managing API keys.

Having the clients send API key within the HTTP header is convenient to handle. We can quickly check the HTTP request header and decide whether to allow or deny the request.

As a prerequisite you should be familiar writing front controller plugins. Let's write a front controller plugin that does the following:


How To Get Broadcom BCM4312 Wireless To Work In Fedora

Submitted by sudheer on Thu, 02/18/2010 - 17:05

Broadcom provides Linux drivers for most of its wireless chipsets. Prior to the release of official drivers from Broadcom, the open source software community had various workarounds. Some people used Windows drivers through ndiswrapper. Others used fwcutter.

The RPM Fusion Repository for Fedora offers the Broadcom wireless drivers from its repositories. You can install them using yum.

Step 1: Enable the RPM Fusion repositories

 
rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm


How To Play Music And Video In Fedora

Submitted by sudheer on Thu, 02/18/2010 - 15:49

How to play music and video(MP3, MPEG, AVI, FLV and other multimedia files) in Fedora 12?

Due to patent encumbrance codecs required to play some proprietary file formats are not shipped with Fedora. Codecs are the programs that encode and decode digital data. These codecs are available in the rpmfusion repository for Fedora. You will be able to play mpeg, mp3, avi, flv and other commonly used file formats for music and video. You will require an Internet connection to download and install these packages on your computer.

video movie playing in gnome mplayer fedora 12 screenshot

Let us get started. Are you ready?


Concluding The Bangalore PHP User Group Meeting - January 30 2010

Submitted by sudheer on Mon, 02/01/2010 - 22:54

Last Saturday, the Bangalore PHP User Group conducted a meeting. The venue was same as the last time, Microsoft office, Bangalore! The topic of the meeting was Framework Shootout. The frameworks represented were:


Syndicate content