VirtualBox, Debian, and Headset support

Posted on by

I’m pleased to announce after much reading and fidgeting I’ve finally got a headset working through VirtualBox guests.

I started with a simple analog headset and MIC but didn’t have much luck with the settings and sound quality. I noticed in VirtualBox it looked seamless to add virtual USB devices so I decided to bite the bullet and buy a USB headset.

I’ve tested with Skype and Webex using a Logitech H360 with the Debian OS. O, and I cannot forget PulseAudio. It’s arguably the greatest leap for sound interoperability in Linux, ever.

Google Transparency Report

Posted on by

Google has a tracking system called “Government Requests”. It records the scope and type of content countries submit to Google for removal from their services, in some sense, our internet (seemingly quite often too). A few days ago Google released data from January through June of this year. I commend Google for going to such lengths and having the shear courage in interest of transparency – you’re doing it right.

In many cases the requests are probably legitimate and imply something illegal has taken place, will, or the data is in violation of someones inalienable rights. However, I have a hard time believing *most* requests fall under that umbrella. Especially, when generalized reasons are common, such as “Other” or “Privacy and Security”.

The data isn’t user, company, entity, or content centric. Since it’s broken into 6 month chunks you can’t really glean a specific moment in time or incident. If not for that, it may help explain a few of these requests. Google doesn’t actually comply with all requests. I’m foggy on the specific details but from what I gather it needs to violate ToS or local law. Anyhow, Google knows what the content is and I’m sure they have an assortment of people critiquing the logic on whether to remove or not.

The variation of reasons, products, and countries are across the board. This represents 2 six month chunks or 1 years worth of data.

mysql> select reason, sum(num_items_requested) num_items_requested, count(distinct country) unique_countries, group_concat(distinct country_code order by country_code) countries  from RequestByProductAndReason group by reason order by sum(num_items_requested) desc;
+----------------------+---------------------+------------------+-------------------------------------------------------+
| reason               | num_items_requested | unique_countries | countries                                             |
+----------------------+---------------------+------------------+-------------------------------------------------------+
| Other                |               97153 |               17 | AR,AU,BR,CA,DE,ES,GB,IN,IT,JP,KR,LY,NO,RU,TR,TW,US    |
| Privacy and Security |               33161 |               17 | AR,AU,BR,CA,DE,ES,FR,GB,IN,IT,JP,KR,LY,NL,PL,TR,US    |
| Copyright            |               11664 |                7 | BR,DE,GB,IT,TR,TW,US                                  |
| Defamation           |                6875 |               18 | AR,AU,BR,CA,CH,DE,ES,FR,GB,IN,IT,JP,KR,NL,PL,TR,TW,US |
| Government Criticism |                 572 |                7 | BR,DE,IN,IT,TH,TR,US                                  |
| Hate Speech          |                 508 |                8 | AU,BR,DE,FR,GB,IN,IT,US                               |
| Pornography          |                 187 |                9 | AU,BR,DE,ES,FR,IN,IT,NL,TR                            |
| National Security    |                 154 |                3 | GB,IN,US                                              |
| Impersonation        |                 141 |                6 | BR,FR,IN,KR,TR,US                                     |
| Violence             |                  65 |                5 | BR,DE,GB,IT,US                                        |
| Electoral Law        |                  36 |                2 | BR,TW                                                 |
+----------------------+---------------------+------------------+-------------------------------------------------------+

mysql> select product, sum(num_items_requested) num_items_requested, count(distinct country) unique_countries, group_concat(distinct country_code order by country_code) countries  from RequestByProductAndReason group by product order by sum(num_items_requested) desc;
+------------------------------------------+---------------------+------------------+-------------------------------------------------------+
| product                                  | num_items_requested | unique_countries | countries                                             |
+------------------------------------------+---------------------+------------------+-------------------------------------------------------+
| Google AdWords                           |               95862 |               10 | AR,BR,DE,ES,GB,KR,NO,TR,TW,US                         |
| Web Search                               |               36373 |               17 | AR,AU,BR,CA,CH,DE,ES,FR,GB,IN,IT,JP,KR,PL,RU,TR,US    |
| Picasa Web Albums                        |               11585 |                4 | BR,ES,IN,US                                           |
| YouTube                                  |                3205 |               18 | AU,BR,CA,DE,ES,FR,GB,IN,IT,JP,KR,LY,NL,PL,TH,TR,TW,US |
| Google Groups                            |                1671 |                4 | BR,FR,IT,US                                           |
| orkut                                    |                1036 |                3 | BR,IN,KR                                              |
| Blogger                                  |                 495 |               16 | AR,BR,CA,CH,DE,ES,FR,GB,IN,IT,KR,NL,PL,TR,TW,US       |
| Google Images                            |                 160 |                6 | AR,BR,DE,GB,IN,US                                     |
| Gmail                                    |                  61 |                8 | BR,CA,DE,ES,FR,IT,TW,US                               |
| Google Videos                            |                  22 |                3 | GB,IT,US                                              |
| Android Market                           |                  10 |                1 | KR                                                    |
| Google Earth, Google Maps, and Panoramio |                   9 |                6 | DE,FR,GB,IN,TR,US                                     |
| Google Places                            |                   8 |                1 | BR                                                    |
| Google Docs                              |                   5 |                1 | FR                                                    |
| Google Profiles                          |                   4 |                2 | IN,KR                                                 |
| Google Sites                             |                   3 |                3 | BR,DE,TW                                              |
| Street View                              |                   3 |                2 | DE,GB                                                 |
| Google App Engine                        |                   1 |                1 | KR                                                    |
| Google Books                             |                   1 |                1 | US                                                    |
| Textcube                                 |                   1 |                1 | KR                                                    |
| Web Search: Autocomplete                 |                   1 |                1 | IT                                                    |
+------------------------------------------+---------------------+------------------+-------------------------------------------------------+

mysql> select country, sum(num_items_requested) num_items_requested  from RequestByProductAndReason group by country order by sum(num_items_requested) desc;
+----------------+---------------------+
| country        | num_items_requested |
+----------------+---------------------+
| United Kingdom |               93851 |
| South Korea    |               32798 |
| Brazil         |               13052 |
| Germany        |                4337 |
| United States  |                2178 |
| Norway         |                1814 |
| India          |                 640 |
| France         |                 366 |
| Turkey         |                 288 |
| Thailand       |                 268 |
| Italy          |                 211 |
| Libya          |                 203 |
| Argentina      |                 132 |
| Taiwan         |                 115 |
| Poland         |                  72 |
| Spain          |                  63 |
| Japan          |                  38 |
| Canada         |                  36 |
| Switzerland    |                  18 |
| Netherlands    |                  16 |
| Australia      |                  10 |
| Russia         |                  10 |
+----------------+---------------------+

I was particularly interested …

  • with the “Government Criticism” -> “Youtube” request by United States.
  • with the “Privacy and Security” -> “Google Profile” request by South Korea.
  • with the “Other” -> “Android Market” request by South Korea.
  • why Norway and Russia always use the reason “Other”.
  • why Thailand always uses the reason or seeks out “Government Criticism”.
  • why Thailand and Libya only seek out “Youtube” content.

Moving onto a similar data set. This shows the percentage of actual data fully or partially removed for each request. This includes roughly 2 years worth of data.

The following result set shows the success rate average for each request for a given 6 month chunk. In the last year or so, of all the items requested to be removed, about 65% percent were fully or partially removed. If you take out the outliers hovering at or below 10 percent success rate you can see the numbers are much higher.

mysql> select date_ending, avg(percent_removal_complied) avg_percent_removed from RemovalRequests group by date_ending order by date_ending desc;
+-------------+---------------------+
| date_ending | avg_percent_removed |
+-------------+---------------------+
| 2011-06-30  |             64.1667 |
| 2010-12-31  |             67.3235 |
| 2010-06-30  |             68.2500 |
| 2009-12-31  |             52.3721 |
+-------------+---------------------+

mysql> select date_ending, avg(percent_removal_complied) avg_percent_removed from RemovalRequests where percent_removal_complied>10 group by date_ending order by date_ending desc;
+-------------+---------------------+
| date_ending | avg_percent_removed |
+-------------+---------------------+
| 2011-06-30  |             77.0000 |
| 2010-12-31  |             84.7778 |
| 2010-06-30  |             87.7500 |
| 2009-12-31  |             72.6452 |
+-------------+---------------------+

This last table shows countries with less than a 25% removal success rate. Which, in theory, would imply they need to work on their censorship logic. Though admittedly, Google doesn’t indicate the number if it’s less than 10 making it rather difficult, inconclusive at best, to say if they have bad censorship logic or if they had the misfortune of being rejected a couple times.

mysql> select date_ending, country, if(num_removal_requests=10,'<10',num_removal_requests) num_removal_requests, if(num_items_requested=10,'<10',num_items_requested) num_items_requested,  percent_removal_complied   from RemovalRequests where percent_removal_complied<=25 and num_removal_requests>1 order by date_ending desc;
+-------------+-------------------+----------------------+---------------------+--------------------------+
| date_ending | country           | num_removal_requests | num_items_requested | percent_removal_complied |
+-------------+-------------------+----------------------+---------------------+--------------------------+
| 2011-06-30  | Taiwan            | 69                   | 115                 |                       12 |
| 2011-06-30  | Colombia          | <10                  | <10                 |                        0 |
| 2011-06-30  | Indonesia         | <10                  | <10                 |                        0 |
| 2011-06-30  | Ireland           | <10                  | <10                 |                        0 |
| 2011-06-30  | Israel            | <10                  | <10                 |                       25 |
| 2011-06-30  | Libya             | <10                  | <10                 |                        0 |
| 2011-06-30  | Malaysia          | <10                  | <10                 |                        0 |
| 2011-06-30  | Pakistan          | <10                  | <10                 |                        0 |
| 2010-12-31  | Belgium           | <10                  | <10                 |                        0 |
| 2010-12-31  | India             | 67                   | 282                 |                       22 |
| 2010-12-31  | Malta             | <10                  | <10                 |                        0 |
| 2010-12-31  | Mexico            | <10                  | <10                 |                        0 |
| 2010-12-31  | Norway            | <10                  | <10                 |                        0 |
| 2010-12-31  | Pakistan          | <10                  | <10                 |                        0 |
| 2010-12-31  | Singapore         | <10                  | <10                 |                        0 |
| 2010-12-31  | Taiwan            | <10                  | <10                 |                       25 |
| 2010-12-31  | Vietnam           | <10                  | <10                 |                        0 |
| 2010-06-30  | Taiwan            | 11                   | 12                  |                        0 |
| 2010-06-30  | Sweden            | <10                  | <10                 |                        0 |
| 2010-06-30  | Cyprus            | <10                  | <10                 |                        0 |
| 2010-06-30  | Kazakhstan        | <10                  | <10                 |                        0 |
| 2010-06-30  | Macedonia [FYROM] | <10                  | <10                 |                        0 |
| 2010-06-30  | Mexico            | <10                  | <10                 |                        0 |
| 2010-06-30  | Russia            | <10                  | <10                 |                        0 |
| 2009-12-31  | Belgium           | <10                  | 0                   |                        0 |
| 2009-12-31  | Colombia          | <10                  | 0                   |                        0 |
| 2009-12-31  | Israel            | <10                  | 0                   |                       20 |
| 2009-12-31  | Cambodia          | <10                  | 0                   |                        0 |
| 2009-12-31  | Lithuania         | <10                  | 0                   |                        0 |
| 2009-12-31  | Macedonia [FYROM] | <10                  | 0                   |                        0 |
| 2009-12-31  | Malaysia          | <10                  | 0                   |                        0 |
| 2009-12-31  | New Zealand       | <10                  | 0                   |                        0 |
| 2009-12-31  | Peru              | <10                  | 0                   |                        0 |
| 2009-12-31  | Pakistan          | <10                  | 0                   |                        0 |
| 2009-12-31  | Sweden            | <10                  | 0                   |                        0 |
| 2009-12-31  | Armenia           | <10                  | 0                   |                        0 |
+-------------+-------------------+----------------------+---------------------+--------------------------+

This information definitely sparks thought. Arguably, although there are plenty of organizations, working groups, etc. influenced by the government, companies still own the internet. What boat we'll be in 10-25 years from now is anyone's guess. It just goes to show that company and government policy is extremely important and it's something that cannot be taken lightly.

I would encourage everyone to take a peak at the following links and data for additional context.

Here's the Google blog entry.
Here's the transparency report homepage.
Here's the full breakdown of the United States (as seen in the above screen shot).

Fun with USGS data – “Did you feel it?”

Posted on by

Remember the 5.8 earthquake that rocked the eastern seaboard August 23, 2011? The epicenter was about 4 miles southwest of Mineral, VA. Well, it was the first earthquake I had ever felt and I wasn’t about to pass on the opportunity of reporting it. Especially, being 350 miles away here in South Carolina.

Not 10 minutes later I went to the USGS’s website and submitted my report via their “Did you feel it” page. Then I waited for a couple hours for others to do the same.

Here’s the image I created not long after.

This is about 3 hours after the first shock illustrating a more than 2,500 reports. Now, if we look at the data from today, there were in total more than 8,500 reports. Here’s that image.

I’m somewhat skeptic. There are even reports west of the Mississippi River. I wonder how many people really felt it, and how many just want to say they did? On USGS’s website, it does have a severity field and some of these are less than 1 yet greater than 0. Regardless, you can clearly see its affects were far reaching.

How to monitor the network for mobile devices

Posted on by

I’ve recently noticed more and more chatter about people wanting to monitor their network for mobile devices. Just for shear existence, not necessarily any output data. Monitoring app and security relevant data from a smartphone is another ball of wax deserving its own post/book.

One such possible solution involves collecting MAC addresses on your network. A handy tool that does just that is called arpwatch.

Here’s the low down from the man page.

DESCRIPTION
Arpwatch keeps track for ethernet/ip address pairings. It syslogs activity and reports certain changes via email. Arpwatch uses pcap(3) to listen for arp packets on a local ethernet interface.

There’s a builtin option with arpwatch to alert you when a new IP/MAC is detected but for our purposes we want to compare the arpwatch data with other information and then do something with it.

You can then take these MAC addresses and cross-reference with the IEEE OUI data. You can download the data from here.

- Create the database and tables for ArpWatch. Feel free to tweak the schema DDL as you see fit.

mysql> create database ArpWatch;
mysql> use ArpWatch;
mysql> show create table ArpWatchData \G
*************************** 1. row ***************************
       Table: ArpWatchData
Create Table: CREATE TABLE `ArpWatchData` (
  `time_detected` datetime NOT NULL,
  `ip_address` varchar(32) NOT NULL,
  `mac` varchar(32) NOT NULL,
  `name` varchar(128) NOT NULL,
  UNIQUE KEY `my_arp_key` (`time_detected`,`ip_address`,`mac`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> show create table OUIData \G
*************************** 1. row ***************************
       Table: OUIData
Create Table: CREATE TABLE `OUIData` (
  `hex_mac` varchar(16) NOT NULL,
  `oui` varchar(512) NOT NULL,
  PRIMARY KEY  (`hex_mac`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

- Load the OUI MAC data from our beloved IEEE friends (note, IEEE updates this data everyday).

# wget http://standards.ieee.org/develop/regauth/oui/oui.txt
# perl -ne 'if ($_ =~ /\(hex\)/ ) { $_ =~ s/[^[:ascii:]]//g; $_ =~ s/\(hex\)/\t/g; my ($hex_mac,$oui) = ($_) =~ /^([A-F0-9-]+).(.*)/; $oui =~ s/^\s+//g; $oui =~ s/"//g; print "REPLACE INTO ArpWatch.OUIData (hex_mac,oui) VALUES (\"$hex_mac\",\"$oui\");\n"}' oui.txt  | mysql -u your_db_username -p -D ArpWatch

The above one-liners are rather simplistic and I just needed something quick and dirty. Ideally this would be placed in a script and cron to update the database on a regular basis. In essence, all we’re doing here is downloading the MAC data from IEEE, removing any non-ascii characters, extracting the first 3 hex values and organization, print out the INSERT/REPLACE statement, and lastly pipe that to MySQL. It should take merely a few seconds.

- Initially, if you already have some Arpwatch data, load it into the ArpWatchData table. Otherwise, “tail” the file with your program for new data then insert accordingly. Take the following script, which loads arpwatch .dat file for example.

#!/usr/bin/perl

use strict;
use DBI;

my ($username) = 'your_db_username';
my ($password) = 'your_db_password';
my ($dbh) = DBI->connect("DBI:mysql:ArpWatch", $username, $password) || die "Could not connect to database: $DBI::errstr";
my ($arp_file) = '/var/lib/arpwatch/eth0.dat';

open(ARP_FILE,$arp_file);

while(<ARP_FILE>) {

        chomp($_);

        # get basic information from Arpwatch dat file
        my ($mac,$ip_address,$time,$name) = $_ =~ /^([a-f0-9:]+)\s+([0-9.]+)\s+([0-9]+)\s+([^ ]+)/;

        # convert epoch to local
        my ($sec,$min,$hour,$day,$month,$year) = (localtime($time))[0,1,2,3,4,5,6];

        # remove spaces from beginning of hostname/name
        $name =~ s/\s//g;

        # make time conform to standard
        my ($formatted_time) = sprintf('%04d-%02d-%02d %02d:%02d:%02d',($year+1900),($month+1),$day,$hour,$min,$sec);

        # make MAC conform to standard
        my ($formatted_mac);
        my (@hexes) = split(/:/,$mac);
        foreach my $hex (@hexes) {
                ($hex) =~ s/^([a-f0-9])$/0$1/;
                $formatted_mac .= $hex . ':';
        }
        $formatted_mac = uc($formatted_mac);
        $formatted_mac =~ s/:$//;

        # print/insert the data. use placeholders!!!
        print "$formatted_time,$ip_address,$formatted_mac,$name\n";
        $dbh->do("REPLACE INTO ArpWatch.ArpWatchData (time_detected,ip_address,mac,name) VALUES (?,?,?,?)",undef,$formatted_time,$ip_address,$formatted_mac,$name);
}

close(ARP_FILE);

- Here’s where we start poking around for MAC addresses, hopefully, related to mobile phones. This isn’t always so cut and dry. As you start evaluating entries within your own network you will began building and refining your where clause or alerting solution. Where permitted, and given you have a valid IP associated with the MAC, you could invoke a port scan or, try probing the device for additional context, such as a service that only listens on mobile phones. I’m sure there are many other ways to (in)validate as well (ie. Bluetooth maybe?). Chances are, it will never be a fool proof (ie. someone changes their MAC) but it should be sufficient to satisfy the 80-20 rule. At very least, it will give you a very good idea of what you and your network is up against.

The following query illustrates aggregated Arpwatch data for my home network for the last year.

mysql> select max(time_detected) last_detected, oui, group_concat(distinct substring_index(mac,':',3)) ou_indentifiers, count(distinct mac) unique_devices from ArpWatchData awd left join OUIData od on(od.hex_mac=replace(substring_index(mac,':',3),':','-')) where (name rlike '(iphone|android_)' or oui rlike '^(HTC Corporation|RIM|Motorola Mobility, Inc\.)$') group by oui order by oui;
+---------------------+-------------------------+----------------------------------------------+----------------+
| last_detected       | oui                     | ou_indentifiers                              | unique_devices |
+---------------------+-------------------------+----------------------------------------------+----------------+
| 2010-12-10 20:09:39 | Apple                   | 7C:C5:37                                     |              1 |
| 2011-05-30 10:33:33 | Apple, Inc              | 00:26:B0                                     |              1 |
| 2011-05-09 22:10:40 | EPIGRAM, INC.           | 00:90:4C                                     |              1 |
| 2011-10-08 23:50:44 | HTC Corporation         | 00:23:76,7C:61:93,38:E7:D8,90:21:55,F8:DB:7F |             14 |
| 2011-05-27 19:40:14 | Huawei Device Co., Ltd  | 30:87:30                                     |              1 |
| 2011-05-31 17:52:00 | Motorola Mobility, Inc. | F8:7B:7A,40:FC:89                            |              6 |
| 2011-05-30 12:42:07 | RIM                     | 3C:74:37,30:69:4B,CC:55:AD,30:7C:30,2C:A8:35 |              5 |
+---------------------+-------------------------+----------------------------------------------+----------------+
7 rows in set (0.00 sec)

Forget the resume, application, and interview – just start working

Posted on by

I would hate to be a recruiter or hiring manager right now. Imagine having the responsibility to siphon through the hundreds and even thousands of resumes and interviews for a single position, knowing good and well another person out there is better suited for the job. It would be incredibly naive and vain to think otherwise. Here’s my attempt to make their job a bit easier and get you a job (maybe, if my logic holds true).

The other day, it was incredibly nice outside. So nice, I couldn’t help but spiff up for the occasion. I shaved (typically, I sport the five o’clock shadow), put on some khaki slacks, and then threw on a sleek blue-grey-business-causal-friday shirt. I didn’t have any particular reason, other than it was nice outside and I needed to get my car battery replaced.

As I paraded around Walmart waiting on the automotive guys to replace my battery, an elderly lady came up to me and asked where the mirrors were. Oddly enough, moments before I was checking out the clocks, and knew exactly where the mirrors were. I confusedly exclaimed, “Umm, see that pallet of DVDS?, (as I pointed) take a right, they’re two rows down on the left.” She said, “Thank you son” and carried on her merry way. Another lady not too far way, whom did work there, looked over at me and asked “I’ve never seen you before, are you new?”. I then proceeded to tell here that I didn’t work there and she chuckled.

I was hungry and needed to kill some more time. My default answer is usually Sushi. After enjoying my meal I was off to enjoy the rest of my day. Not only do I love Sushi, I’m also an avid toothpick enthusiast after which. As I was leaving, aimlessly looking for the toothpick dispenser at the hostess counter, a mother and daughter walked through the entrance of the restaurant. Unfortunately, they were out of toothpicks but as I turned around the mom asked if they could be seated outside. “Most certainly, I don’t see why not, I just did!?”, I said. She uttered a quick laugh and obviously got the hint I didn’t work there.

But this got me thinking. What if, you just started walking into businesses helping customers acting as if you work there. I didn’t even try and people thought I worked there! Call me crazy but I’m inclined to say, someone, somewhere will appreciate your candid and unorthodox method, willingness to be out of your element, appreciate the help, and give you a job. Eventually, you would probably be told to leave at least once. It seems worth it though, if not for the job, at least it would make for a good story for Thanksgiving or Christmas dinner.

Here are a few more ideas. Whatever your choice of business, do something you enjoy and don’t mind doing it for free.

- Like/love RC cars or flying kites? Check out the hobby store.
- Like/love playing guitar or the piano? Check out the music store.
- Like/love home interior/scrap booking? Check out home good stores.
- Like/love computers? Offer assistance to the elderly at a retirement home or local library.
- Like/love fitness and working out? Read up all about fitness, get fit if you aren’t, then go mentor at the local fitness center.
- You get the picture

A wise man once said …

“If you build it, he will come”.
- Shoeless Joe Jackson

Monitoring your Gmail account activity

Posted on by

I can’t quite recall if I’ve ever noticed this before but Gmail has a “Last account activity” section in the footer on the right.

If you click the details link a new window opens.

“Last account activity shows you information about recent activity in your mail. Recent activity includes any time that your mail was accessed using a regular web browser, a POP1 client, a mobile device, etc. We’ll list the IP address that accessed your mail, the associated location, as well as the time and date.”

Here’s more information about it.

The alerting preferences are meager: show an alert for unusual activity or don’t.

While we’re at it, might I pose a suggestion. Google should A) enable activity tracking for other applications (Docs, Calendar, Finance, Photos, etc) and B) have an API or a secure forwarding mechanism to enable another program to monitor.

Although Google does in fact monitor Gmail for “suspicious activity”, defining the algorithm is highly subjective. Ones intentions fluctuate from account-to-account or application(s) you wish to monitor. For example, I want to see Gmail login (or logout) activity for the last 3 years and cross-reference with Geo data for non-US IP addresses or a blacklist. Or, what if I want to see when a Doc is shared, deleted, or modified.

I imagine this would energize many cloud based initiatives, products, and consumers, particularly in regards to Chromebook, compliance, and security topics in general.

Anyhow, pretty cool.

Horry County Police Department booking data

Posted on by

Like many police departments across the country, the Horry County Police Department in Horry County, SC publishes booking data online. You can search by last name and the booking or release date. This sparked my data-mining interest so I decided to engage.

They use an ASP page which accepts the following POST parameters:
InputLastName
InputBookingDate
InputReleaseDate

Easy enough.

- I hacked up a loop to generate back-fill dates for the past ~year
- Used LWP::UserAgent to issue the POST
- Extracted and normalized corresponding meta-data from HTML
- Stored the data in MySQL

Available data includes time booked, mugshot, name, bail, charge, address, gender, age, etc.

I’m not going to disclose anything earth shattering but it’s interesting data nonetheless to analyze. Not to mention, I wonder who would be interested in such data …

“the” Top 10

Age

Gender

Stay away from people with these names

States

How to syslog data from a Splunk server

Posted on by

This is a brief overview on how to syslog logs originating from both the Splunk daemon itself and devices forwarding data to the Splunk server. A few use cases include backup, compliance, failover, and other data-mining engagements.

Files Involved

There are three configuration files you need to modify in order to forward data from a Splunk server: outputs.conf, props.conf, and transforms.conf. These files are located in $SPLUNK_HOME/etc/system/default/. To set custom configurations (recommended), place the accompanying .conf files in $SPLUNK_HOME/etc/system/local/.

Forwarders require outputs.conf; whereas non-forwarding Splunk instances (indexers) do not use. The file is used to determine how the forwarder sends data to receiving Splunk instances, either indexers or other forwarders, and/or non-Splunk systems.

The mechanics (file precedence, privileges, CLI, etc) are beyond the scope of this document but generally speaking, settings from within the props.conf and transform.conf files work in unison to perform a given function. Many of the actions in props.conf involve a corresponding transforms.conf configuration. At a high-level, props.conf says what rules are applied to an event and when they are applied and transforms.conf actually defines those rules.

Settings

Here is a template which should be sufficient to get things started. You will need to replace the destination address of 127.0.0.1 with the IP address of your syslog server. You can also change the port or protocol if need be.

# outputs.conf
[syslog:dest_syslog_srv]
server = 127.0.0.1:514
type = udp
disabled = false
sendCookedData = false

# props.conf
[host::*]
TRANSFORMS = send_to_dest_syslog_srv

# transforms.conf
[send_to_dest_syslog_srv]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = dest_syslog_srv
WRITE_META = false

Notes

You can enable configuration changes made to transforms.conf, props.conf, and outputs.conf by typing the following search string in Splunk Web

| extract reload=T

or you can take the traditional route

/opt/splunk/bin/splunk restart