We’ll define some global variables also (I know not a recommended practice, but given the time constraint, it was a trade off we all have to take) # Query to find the max record processed so far get_max_query = 'SELECT COALESCE(max(itunes_episode),0) FROM tasteofindia.posts ' # Query template to insert values in any table query_string = 'INSERT INTO tasteofindia.') # List comprehension on the result of map operation on records )] return records

Imports: The following lines will import the required modules and objects from data_parser import get_soup, parse_record, store_tags from db_connect import get_connection, get_max_records,execute_query from import BlockingSchedulerĢ. With approach shared above, let’s get you started with the implementation!įinally let’s build the script connecting all the pieces together… 1. Also, since no environment specifications were given, I decided to build all of this using two docker ( ) containers, one for python and one for postgres. Library psycopg2 ( ) to support postgres ( ) (don’t think it needs an explanation, but didn’t want to use wrappers like SQLAlchemy ( )) 5. Library apscheduler ( ) to schedule task once a day (a simple library to start with, airflow ( )/ azkaban ( ) would’ve been an overkill) 4.

No particular reason, except ease of usage and familiarity and obviously an insanely popular open source support. The relational database I chose was postgres ( ). Since it’s ultimately xml-based content, I decided to use the ever-reliable BeautifulSoup Library ( ) 2. The restriction of RSS parser libraries, basically expected me to write my own parser. All other decisions were pretty straight-forward given the time constraint of 1 day, leaving 3–4 hours for this work (assuming 8–9 hours given to the day job).

#3 above basically meant I had to implement the RSS parser on my own. Obviously I decided to use python 3.7, with the end of life support coming for python 2.7.

Python rss feed reader