G’day to Big Data Beard from the sunny land Down Under. Happy to be a contributor to my American mate’s fun blog!
As a long time Dell EMC’er, Canuck, Aussie, and a Splunk Architect, looking forward to adding some sweet-as and very valuable contributions.
So here goes… Had some good times last couple weeks working with a awesome developer, Crest Data Systems, who wrote, and maintains our XtremIO and Isilon Apps for Splunk Enterprise.
Dell EMC has had an Isilon App for Splunk Enterprise for quite some time. This app has done a great job of collecting and reporting on Isilon array configuration and performance data. Due to massive customer demand, we have had the app upgraded to do some “basic” file system auditing. This is not intended as a full replacement for commercial products that may solely focus on auditing solutions, but more a solution that gets customers a great “starter” Isilon auditing solution. Assuming you already own Splunk, it’s free, as the app is a no charge download! That being said, there is nothing stopping enterprising individuals to take what we started, and expand it into a really cool and customised auditing solution for their environment. That’s one cool part of Splunk.
For those of you who are new to Splunk, you can get this app from http://splunkbase.splunk.com. The app comes in two parts – first being the TA (technology add-on) that has collection scripts that make REST calls to the Isilon array to collect performance and configuration data. The second part is the app it self which provides all the awesome visualisations.
The app is fairly easy to setup. Deploy the TA and app in your Splunk environment, this varies depending on whether you have a small single server deployment or a more complex and scalable distributed/clustered environment. This is covered well in the app install instructions, give me a shout if you have questions. I’ve set it up many times.
A diagram below of a simple setup which collects from two Isilon clusters. First, I will answer some questions on deployment. The Isilon configuration and performance data is collected via REST API in a pull methodology from the Splunk forwarder or where ever the TA is installed.
A series of REST modular inputs are setup on the Isilon TA that periodically poll the Isilon array or arrays for the REST data. They look something like below and can also be tweaked and manipulated individually.
A question I get asked is what user credentials are required? Of course root works, but you wouldn’t want to deploy that way in a production environment. Instead I created a read-only Isilon user that has the ability to log into gui and collect auth, event, and statistics. Here are the privileges I provided to the user.
Also, an important item to note. When you setup the Splunk app, do NOT use an Isilon Smart connect IP. This causes issues with the REST calls and authentication. Use a single node IP.
On the auditing side, the syslog data is received from the Isilon nodes via syslog forwarding. You have to enable the Isilon to forward data to the Splunk forwarder. This has to be done via SSH, and root access to the Isilon is required, but it’s a one time deal. Login as root and edit /etc/mcp/override/syslog.conf. You need to add in the lines below, you only need the IP address of the Splunk receiving forwarder, one per each of the three sections. In this example, I am forwarding to two separate Splunk environments for testing purposes. Once you save the file, that’s it. The Isilon will ensure all nodes have the same copy of the conf file. You should see events at your Splunk forwarder. Make sure to enable receiving on port 514 on your fowarder. The Isilon will forward events from all nodes.
So what goodness do you get from all this? Have a look at this cool dashboard, complete with shiny new Dell EMC logo!
On the FS Audit dashboard under Security menu, you get a quick view of whats going on. Top accessed files, top client IP addresses, top failed reasons. You have the ability to sort based on Isilon Cluster name and/or users on the Isilon array. The TA currently queries both the local authentication provider as well as all AD authentications providers. It also does a mapping of the AD users SIDs to friendly names to make viewing easier. AD users are shown with the AD prefix to make selections easy, especially when similar names users exist across multiple domains.
If you need some more detailed information, you can head over to the next dashboard, which is the audit logs search facility. In here, you can do a detailed search on specific user activities, filtered by Isilon cluster, user name, event, action, or string.
Here you can see the pesky user “VCE/Dean” who seems to be creating, opening, deleting some files with “splunk” in them 🙂
You can even see the source IP address that pesky user is connecting from. If the user is particularly active, you could continue on to filter by action (OPEN, CLOSE, READ, WRITE, DELETE, LOGOFF), or by event (SUCCESS, FAILED).
Lastly, what if you want to see who is trying to get into your cluster as root? In the last dashboard, authentication and privilege activities, you can see just that. Here you can see someone is trying to get into the Isilon cluster as user root repeatedly and unsuccessfully. The panel on the right indicates the IP address the user is trying to SSH in from. Seems like a similar address to that pesky user “VCE/Dean”, hmmmmm…
Well, that’s a quick look at the new updates to the Dell EMC Isilon app for Splunk enterpeise. If you want to see it live, have a look at this recorded demo:
That’s about all for now, hope you all enjoyed this post, for those in Orlando for Conf, drop by my session (along with Splunk SE extraordinaire Simon O’Brien). It’s on Wednesday at 11am in Dolphin A1 and is called: Bucket Diversity: Choosing Your Search Mate Wisely. You can learn some fun stuff about sizing/designing Splunk buckets and learn a little about Australia as well 🙂
Make sure you come by and say g’day!!
[…] Auditing Excitement With Isilon Enabled By Splunk […]
Dear Dean Jackson, Thank you for the detailed steps. We have got similar requirement from client to integrate Isilon with Splunk. However we having few issues.
1) Isilon able to forward logs to Splunk using root, but not any other local user or AD user. Do you think we are missing anything?
2) Both config logs and protocol logs will be forwarded from isilom syslogs to Splunk ,right ?
Reason is, we are able to see config logs in spliunk, but not protocol logs…Much appreciated if you can help me…Thanks
Shoot me an email at brett@BigDataBeard.com and I can help