Skin of Stars » php http://skinofstars.com Machines, Media and Miscellanea Sat, 19 May 2012 11:27:14 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 PHP Script for RSS auto-discovery and OPML file generation http://skinofstars.com/2010/03/php-script-rss-auto-discovery-opml-file/ http://skinofstars.com/2010/03/php-script-rss-auto-discovery-opml-file/#comments Sun, 07 Mar 2010 19:21:40 +0000 Kevin http://skinofstars.com/?p=369 ... Read the remaining 654 words]]> Hey All,

I recently got a reasonable size list of blog URLs. What I wanted was to import all these into a feed reader (via OPML). There seemed to be a lack of conversion scripts for batch URL->find RSS link->feed reader import file (I may be wrong, please let me know if I am :) , so I made one in PHP. I guess this is like an automatic-blogroller. I have just used this as a command line script, I’m not recommending you use this in ‘the wild’ as one might say, I have made little concession to security as I had a trusted list of URLs.

There are basically three steps to this

  1. Take an input file of newline seperated URLs, in my case blogs.
  2. Find (auto-discover) associated RSS feed of each blog URL
  3. Output an OPML file that you can use to import into a feed reader

What it does:

  • Takes a well formed list of newline separated URLs of blogs and turns it into an OPML
  • If the URL source doesn’t contain a <link> to an RSS feed in the head it doesn’t add it to the OPML
  • Detects the <title> and adds that to the OPML text field, or uses the URL if <title> isn’t present

What it doesn’t:

  • Check the RSS feed is validated XML
  • Any other checking really :)
  • Come with any sort of warranty/guarantee

Some of the key functions are from Keith Devens work. Thanks.

Without any further ado, here is the script:

<?php
/*
 * @author @skinofstars Kevin Carmody
 * GPLv3 - http://www.gnu.org/copyleft/gpl.html
 *
 * this is really a command line app with no flags
 * for turning a bunch ofurls into an OPML file
 *
 * 1.takes input file of newline seperated urls, normally blogs
 * 2.finds (autodiscovery) associated rss of each url
 * 3.outputs an OPML file for you to use in a feed reader
 */

// file config
$inputFile = "/path/to/URLlist.txt";
$outputFile = "/path/to/blogroll.opml";

// OPML config
$opmlTitle = "Some Select Blogs";
$opmlOwnerName = "Kevin Carmody";
$opmlOwnerEmail = "kevin@skinofstars.com";

/** no need to edit after this :)  **/
$inHandle = @fopen($inputFile, "r");//read-only
$outHandle = @fopen($outputFile, "a");//append

if ($inHandle && $outHandle) {
	$headerOut = opmlHeader($opmlTitle,$opmlOwnerName,$opmlOwnerEmail);
	fwrite($outHandle,$headerOut);

	while (!feof($inHandle)) {
		$buffer = fgets($inHandle, 4096);
		$source = getFile($buffer);
		$rssURL = getRSSLocation($source, $buffer);
		$rssTitle = htmlentities(getTitleAlt($source));
		if($rssURL){
			if($rssTitle){
				$entryOut = opmlEntry($rssURL,$rssTitle);
				fwrite($outHandle,$entryOut);
			} else {
				$entryOut = opmlEntry($rssURL,$rssURL);
				fwrite($outHandle,$entryOut);
			}
			//echo ".";//uncomment to print a dot to screen on each success, nice for seeing progress
		} else {
			echo "Fail on: ".$buffer;
		}
	}
	$footerOut = opmlFooter();
	fwrite($outHandle,$footerOut);

	fclose($inHandle);
	fclose($outHandle);
} else {
	if(!$inHandle){
		echo 'not got a handle on input file: '.$inputFile."\n";
		die;
	}
	if(!$outHandle){
		echo 'not got a got handle on output file: '.$outputFile."\n";
		die;
	}
}

echo "\nAll done :) \n";

/**
 * basic opml header
 * @param string $opmlTitle
 * @param string $opmlOwnerName
 * @param string $opmlOwnerEmail
 * @return string
 */
function opmlHeader($opmlTitle,$opmlOwnerName,$opmlOwnerEmail){
	$oheader = "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\n"
	."<opml version=\"1.1\">\n"
	."	<head>\n"
	."		<title>".$opmlTitle."</title>\n"
	."		<dateCreated>".date("r")."</dateCreated>\n"
	."		<ownerName>".$opmlOwnerName."</ownerName>\n"
	."		<ownerEmail>".$opmlOwnerEmail."</ownerEmail>\n"
	."		</head>\n"
	."	<body>\n";
	return $oheader;
}

/**
 * just returns a test footer
 * @return string
 */
function opmlFooter(){
	$ofooter = "  </body>\n"
	."</opml>";
	return $ofooter;
}

/**
 * creates an XML entry for the OPML file
 * @param string $feedURL
 * @param string $feedTitle
 * @return string
 */
function opmlEntry($feedURL,$feedTitle){
	$outline = "    <outline text=\"".$feedTitle."\" type=\"rss\" xmlUrl=\"".$feedURL."\"/>\n";
	return $outline;
}

/**
 * returns the page title extracted from source
 * @param string $html
 * @return string
 */
function getTitleAlt($html) {
	if (preg_match('/<title>(.*?)<\/title>/is',$html,$found)) {
		$title = $found[1];
		return $title;
	} else {
		return;
	}
}

/**
 * http://keithdevens.com/weblog/archive/2002/Jun/03/RSSAuto-DiscoveryPHP
 * public domain
 */
function getFile($location){
	$ch = curl_init($location);
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
	curl_setopt($ch, CURLOPT_HTTPHEADER, array('Connection: close'));
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	curl_setopt($ch, CURLOPT_TIMEOUT, 15);
	$response = curl_exec($ch);
	curl_close($ch);
	return $response;
}

/**
 * http://keithdevens.com/weblog/archive/2002/Jun/03/RSSAuto-DiscoveryPHP
 * public domain
 */
function getRSSLocation($html, $location){
	if(!$html or !$location){
		return false;
	}else{
		#search through the HTML, save all <link> tags
		# and store each link's attributes in an associative array
		preg_match_all('/<link\s+(.*?)\s*\/?>/si', $html, $matches);
		$links = $matches[1];
		$final_links = array();
		$link_count = count($links);
		for($n=0; $n<$link_count; $n++){
			$attributes = preg_split('/\s+/s', $links[$n]);
			foreach($attributes as $attribute){
				$att = preg_split('/\s*=\s*/s', $attribute, 2);
				if(isset($att[1])){
					$att[1] = preg_replace('/([\'"]?)(.*)\1/', '$2', $att[1]);
					$final_link[strtolower($att[0])] = $att[1];
				}
			}
			$final_links[$n] = $final_link;
		}
		#now figure out which one points to the RSS file
		for($n=0; $n<$link_count; $n++){
			if(strtolower($final_links[$n]['rel']) == 'alternate'){
				if(strtolower($final_links[$n]['type']) == 'application/rss+xml'){
					$href = $final_links[$n]['href'];
				}
				if(!$href and strtolower($final_links[$n]['type']) == 'text/xml'){
					#kludge to make the first version of this still work
					$href = $final_links[$n]['href'];
				}
				if($href){
					if(strstr($href, "http://") !== false){ #if it's absolute
						$full_url = $href;
					}else{ #otherwise, 'absolutize' it
						$url_parts = parse_url($location);
						#only made it work for http:// links. Any problem with this?
						$full_url = "http://$url_parts[host]";
						if(isset($url_parts['port'])){
							$full_url .= ":$url_parts[port]";
						}
						if($href{0} != '/'){ #it's a relative link on the domain
							$full_url .= dirname($url_parts['path']);
							if(substr($full_url, -1) != '/'){
								#if the last character isn't a '/', add it
								$full_url .= '/';
							}
						}
						$full_url .= $href;
					}
					return $full_url;
				}
			}
		}
		return false;
	}
}

Though this was really a one time hit for me it may well be useful to others. Please let me know if you can think of ways to improve it and I will update accordingly.

Thanks,
Kevin

]]> http://skinofstars.com/2010/03/php-script-rss-auto-discovery-opml-file/feed/ 3 Geek Set-Up http://skinofstars.com/2008/04/geek-set-up/ http://skinofstars.com/2008/04/geek-set-up/#comments Thu, 17 Apr 2008 15:59:24 +0000 Kevin ... Read the remaining 15 words]]> I saw this comment on slashdot by http://slashdot.org/~Sancho and it made me chuckle. Please bear in mind that this is a *very* geeky joke.

Linux is ok on the desktop, but for servers, I really prefer FreeBSD.

Sometimes I get tired of that, and use NetBSD for a while. A short stint with NetBSD, Apache, Postgresql, and PHP usually refreshes me long enough so that I can FreeBSD, Apache, Postgresql, and PHP again.

]]> http://skinofstars.com/2008/04/geek-set-up/feed/ 0 Making your own webdev environment on Ubuntu desktop http://skinofstars.com/2008/01/making-your-own-webdev-environment-on-ubuntu-desktop/ http://skinofstars.com/2008/01/making-your-own-webdev-environment-on-ubuntu-desktop/#comments Tue, 01 Jan 2008 12:36:00 +0000 Kevin ... Read the remaining 379 words]]> These machines aren’t all fun fun fun, sometimes you have to use them for some more serious work. In this post I’m going to tell you how to make your own php/mysql web development environment on your own Ubuntu desktop. Once you’ve done this you won’t need to keep uploading files to your server just to check if something works.

Firstly I’d like to point to the post that originally helped me do this at webmasterview.com.

Now, time to open up our trusty terminal. It is possible to do this through synaptics, but its quicker to do it like this:

First thing is to install mysql for databases on our machine. Copy this bit of code into your terminal:

sudo apt-get install mysql-server

Next we need apache2 as our webserver. Here is the code:

sudo apt-get install apache2

Then we need to get php:

sudo apt-get install php5

There are our three basic components. Now we need to hook them together:

sudo apt-get install libapache2-mod-php5

and

sudo apt-get install php5-mysql

Now that’s just grand. We’ve now got all the elements that we need to serve files to the net. The thing is, we’re only doing this so we can work on stuff at home, in which case what we’re really looking for is an easier way to do so. What we need to do is have a public_html folder in our user directory. Perhaps now would be a good time to do that. The next step is making that work on the webserver we’ve just installed. Run this snip of code:

sudo a2enmod userdir

Then restart apache:

sudo /etc/init.d/apache2 force-reload

You should now be able to reach your home folder via http://localhost/~yourusername/

Finally we want to be able to administer our mysql databases. Though there are many good tools for doing this I still go back to the web based phpMyAdmin. Apart from the fact that it’s highly automated (which suits me just fine) it’s also a very common program to find on your paid hosting, so would also be very familiar when you go live. For the quick and dirty way of getting it up and running, download phpMyAdmin and extract to your public_html folder. Personally I rename it to something a little easier. Then rename the file within from config.example.inc.php to config.inc.php and within that file put something in the blowfish_secret field. Then all you have to do is navigate your browser to something like http://localhost/~yourusername/phpMyAdmin

addendum

If your server is going to be visible from the big bad world and this is your everyday machine then you need to turn off that user public_html stuff (it’s insecure for general use!). To do that just enter this instruction

sudo a2dismod userdir

then reload apache as before

]]> http://skinofstars.com/2008/01/making-your-own-webdev-environment-on-ubuntu-desktop/feed/ 2