Using curl and POST requests to automate data retrieval from a website

Problem: A site contains data which is of interest to us. The data is uniformly structured and available for examination by the general public (i.e. no login required) but is not automatically accessible, i.e. viewing it requires repetitive user interaction (such as setting a date range, clicking a ‘submit’ button, etc). In addition, usually the […]

Continue reading


How to make ‘man’ or ‘less’ paged text not disappear from screen after exit?

Problem: Depending on your Linux distribution, the text displayed by the ‘man’ command  (or any other text piped into ‘less’ for that matter) may stay on your screen upon exit from man/less or be cleared from your screen upon exit. Many people find it useful to have the man page text stay on the screen, […]

Continue reading