Sumo Logic to Read Log Data Using Python
Sumo Logic is a cloud-based service that collects, manages, and analyzes log information. NXLog can be configured to send log information to Sumo Logic in syslog format over TCP, or via a custom HTTP endpoint. Additionally, it can too exist configured to ship host metrics via HTTP.
| Note | Sumo Logic offers dissimilar plans for their service including a gratuitous plan. Some of the methods described here may require a paid plan. |
126.1. Using NXLog to collect information
Sumo Logic accepts information from two types of collectors, installed or hosted. Installed collectors are set up by installing agent software provided past Sumo Logic, whereas hosted collectors are used to send data over TCP or HTTP(Due south) from agents similar NXLog. This section presents some scenarios in which using an NXLog agent with a Sumo Logic hosted collector has an advantage over using a Sumo Logic installed collector.
Flexible log processing
NXLog provides more control at the log processing phase earlier events are sent to the Sumo Logic service. This is specially useful since Sumo Logic charges per ingested byte. Although installed collectors support processing rules to drib or keep sure events, these are quite basic compared to the flexibility an NXLog agent provides. When used with NXLog, event data can be manipulated to determine which information to keep, rather than dropping an entire event, too as having full control over the output format. See the topic on Reducing Bandwidth and Data Size for some of the configuration options available when using NXLog.
Back up for a wider range of log sources
Sumo Logic collectors are designed to piece of work with standard log formats primarily stored in apartment files. When logs were non generated by a Sumo Logic supported log source or are not stored in a supported format, such as data stored in a database, network packet captures, or other types of events which are not in a standardized log format, it may not be possible to process such information with the Sumo logic collector. With its vast option of input modules, NXLog can easily fill this gap as it tin be configured to process a wide spectrum of log sources. Additionally, information technology tin likewise process log information using custom scripts with support for Get, Java, Perl, Python and Ruby. Encounter the NXLog documentation on Available Modules for a consummate list of input, processing, and extension modules.
Platform requirements
The Sumo Logic collector is a Java-based agent that requires a substantial amount of resource. NXLog, with its smaller footprint, is a powerful alternative for employ cases where the dependency on Java needs to be eliminated or a less resource intensive agent with more flexibility is preferred. Case in indicate: when installed on a Microsoft Windows 10 system and processing only the Application Windows Event Log channel, information technology was observed that the Sumo Logic collector consumes 200+ MB of memory, compared with 4-6 MB of retention usage by the NXLog agent when tested under the same conditions. See the table beneath for the minimum system requirements for both the NXLog and Sumo Logic agents. Furthermore, the Sumo Logic collector does not back up collecting events in the older Microsoft Windows Event Log format as documented in this Sumo Logic article on Windows 2003 Effect Logs ingestion. NXLog, on the other mitt, specifically supports the Windows XP/2000/2003 result log format through its im_mseventlog input module and can exist installed on systems running such older Microsoft Windows releases.
| NXLog | Sumo Logic Collector | |
|---|---|---|
| Processor Cores | 1 | 1 |
| Memory | sixty MB | 512 MB |
| Disk Space | l MB | 8 GB |
| Pre-requisites | - | Java 1.viii+ |
126.two. Setting upward a hosted collector
For Sumo Logic to receive information over TCP or HTTP(S), a hosted collector must be created from the Sumo Logic spider web interface. To create a hosted collector for data coming from NXLog:
-
Log in to the Sumo Logic web interface.
-
Navigate to Manage Data > Collection.
-
On the Drove tab, click on Add Collector.
-
Choose Hosted Collector, fill in the required details, and click Relieve.
126.three. Setting up the digital certificate
To be able to send data to Sumo Logic, a PEM-encoded DigiCert Certification Authority document is required. This section describes how to prepare the certificate on Linux and Microsoft Windows to be used by NXLog.
126.iii.1. Linux
On Linux, the document needs to exist downloaded from the DigiCert website and then the OpenSSL tool can be used to convert information technology.
-
Download the DigiCert DER encoded certificate:
$ sudo wget -O digicert_ca.der https://world wide web.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt -
Catechumen the certificate to a PEM-encoded certificate:
$ sudo openssl x509 -inform der -in digicert_ca.der -out digicert_ca.crt -
Re-create
digicert_ca.crtto a location accessible by NXLog.
126.three.2. Microsoft Windows
On Microsoft Windows, the document can be exported using the Certificates MMC snap-in.
-
Go to the Windows Showtime menu, type certmgr.msc, and click Enter.
-
Expand Trusted Root Certificatation Regime > Certificates.
-
From the list of certificates, correct click on DigiCert High Assurance EV Root CA and select Open.
-
Go to the Details tab and click the Copy to File… button.
-
The Document Export Wizard opens, click Next.
-
Select Base-64 encoded Ten.509 (.CER) and click Next.
-
Click on the Scan… button and select a location accessible by NXLog.
-
Enter a filename e.grand. digicert_ca.cer and click Save.
-
Click Side by side and then Finish to complete the export.
126.4. Sending logs to Sumo Logic using TCP
Sumo Logic accepts log information every bit syslog messages in IETF (RFC 5424) format and requires data to be sent using TLS v1.2 over TCP. To be able to use this method, a Cloud Syslog Source must be created in Sumo Logic.
-
From the Sumo Logic web interface, navigate to Manage Data > Drove.
-
On the Collection tab, click on Add Source adjacent to the previously created hosted collector.
-
Select Deject Syslog and fill up in the required details. It is recommended to specify a Source Category to brand events from this source easily searchable in Sumo Logic.
-
Click Save to finish creating the syslog source.
-
On cosmos of the new syslog source a dialog is displayed containing the Token, Host, and TCP Port. The NXLog configuration will demand these details for sending log data to Sumo Logic.
For further details on configuring a syslog source, see the Sumo Logic documentation on Cloud Syslog Sources.
| Note | Syslog messages must be compliant with RFC 5424 or they will be dropped by Sumo Logic. Messages over 64KB volition exist truncated. |
Example 659. Sending syslog messages using TLS encryption
In this configuration, NXLog processes a log file using the im_file module, and so generates a syslog bulletin using the xm_syslog module and sends it to Sumo Logic using the om_ssl module. The log data is formatted as JSON using the xm_json module.
The SUMO_TOKEN value needs to exist replaced with an actual Sumo Logic Syslog source token. In this case, processing is done in the output example where the NXLog ID in the structured data of the syslog message is replaced with the token divers by the SUMO_TOKEN constant.
The SUMO_HOST value needs to exist replaced with an actual Sumo Logic Syslog source host URL.
The SUMO_PORT value needs to exist replaced with the correct port for the host defined by the SUMO_HOST abiding.
Output sample
This sample of the syslog header volition be sent to Sumo Logic. The structured data contains the Sumo Logic token as specified in the NXLog configuration.
<13>ane 2020-12-04T11:33:47 NXLog-Ubuntu-1 systemd i - [xxxxxxxxxxxxxxxxxxxx@41123] This sample JSON-formatted event will be sent every bit part of the syslog message to the Sumo Logic service.
{ " EventReceivedTime " : " 2020-12-04 13:37:47 " , " SourceModuleName " : " file " , " SourceModuleType " : " im_file " , " SyslogFacilityValue " : 1, " SyslogFacility " : " USER " , " SyslogSeverityValue " : 5, " SyslogSeverity " : " Notice " , " SeverityValue " : 2, " Severity " : " INFO " , " Hostname " : " NXLog-Ubuntu-1 " , " EventTime " : " 2020-12-04 11:33:47 " , " SourceName " : " systemd " , " ProcessID " : " 1 " , " Message " : " Started Run anacron jobs. " } Example 660. Sending Windows Event Log events as syslog messages
In this configuration, NXLog processes Windows Event Log events using the im_msvistalog module, generates a syslog message using the xm_syslog module, and sends it to Sumo Logic using the om_ssl module. The log data is formatted as JSON using the xm_json module.
The SUMO_TOKEN value needs to be replaced with an actual Sumo Logic Syslog source token. In this example, processing is done in the output instance where the token defined by the SUMO_TOKEN abiding is inserted before the JSON information.
The SUMO_HOST value needs to be replaced with an actual Sumo Logic Syslog source host URL.
The SUMO_PORT value needs to exist replaced with the right port for the host defined by the SUMO_HOST constant.
Output sample
This syslog header sample will be sent to Sumo Logic. The structured data contains the Sumo Logic token equally specified in the NXLog configuration.
<11>1 2020-12-04T11:31:38.031887Z Hopper VBoxNetLwf 0 - [xxxxxxxxxxxxxxxxxxxx@41123] This sample JSON-formatted issue will exist sent as function of the syslog message to the Sumo Logic service.
{ " EventTime " : " 2020-12-07 xi:thirty:nineteen " , " Hostname " : " Hopper " , " Keywords " : " 36028797018963968 " , " EventType " : " ERROR " , " SeverityValue " : 4, " Severity " : " Error " , " EventID " : 12, " SourceName " : " VBoxNetLwf " , " TaskValue " : 0, " RecordNumber " : 44758, " ExecutionProcessID " : 0, " ExecutionThreadID " : 0, " Channel " : " System " , " Message " : " The commuter detected an error on \\ Device \\ VBoxNetLwf. " , " Information " : " \\ Device \\ VBoxNetLwf " , " EventData.Binary " : " 00000C0001000000000000000C0004 " , " EventReceivedTime " : " 2020-12-07 eleven:31:38 " , " SourceModuleName " : " eventlog " , " SourceModuleType " : " im_msvistalog " } 126.5. Sending data to Sumo Logic using HTTPS
Logs and host metrics can be sent to Sumo Logic over HTTP(S) using a unique URL generated for each source. For NXLog to be able to send information over HTTPS, an HTTP Logs & Metrics Source must be created in Sumo Logic.
-
From the Sumo Logic web interface, navigate to Manage Data > Collection
-
On the Collection tab, click on Add together Source side by side to the previously created hosted collector.
-
Select HTTP Logs & Metrics and fill up in the required details. It is recommended to specify a Source Category to make events from this source easily searchable in Sumo Logic.
-
Click Save to finish creating the HTTP source.
-
On creation of the new HTTP source a dialog is displayed containing the HTTP Source Address. The NXLog configuration volition demand this URL for sending log data to Sumo Logic.
For further details on configuring an HTTP source, run across the Sumo Logic documentation on HTTP Logs and Metrics Sources.
126.5.one. Sending log data
Instance 661. Sending logs in batches over HTTPS
In this configuration, NXLog POST log data to Sumo Logic using the om_http module. To send information in batches, the BatchMode directive in the output instance needs to be set to multiline to specify that each log tape should be separated by a new line.
Sumo Logic accepts batched requests up to 1MB of uncompressed data. For simplicity, this case uses the default NXLog batch settings. For further configuration options, see the BatchSize and BatchFlushInterval directives.
Data can be sent to Sumo Logic as patently uncompressed or compressed by the deflate or gzip method. The om_http module supports data pinch based on the zlib compression library (deflate). In the configuration below, the HTTPSSSLCompression directive specifies that compression should be used when sending data to Sumo Logic. If this directive is not specified or set to Imitation, information will be sent uncompressed.
When sending data over HTTP(South), Sumo Logic accepts additional optional headers to configure custom settings related to the log records. For more information, see the Sumo Logic documentation on Supported HTTP Headers. The om_http module supports specifying additional headers by using the AddHeader directive. In the configuration beneath, the X-Sumo-Category header is added with the value my-category.
The SUMO_URL value needs to be replaced with an actual Sumo Logic HTTP source URL.
| Notation | It may have a few minutes for information to be shown in the Sumo Logic spider web interface. If not, see the Sumo Logic documentation on troubleshooting HTTP Sources. |
126.5.2. Sending host metrics
In Sumo Logic terms, host metrics are a gear up of data points that measure out the value of a property over time. For example, host metrics can exist used to monitor the availability and performance of an application, or the resource usage of a host. For more data, see the Overview of Metrics in the Sumo Logic documentation.
Sumo Logic supports metrics in the Graphite, Carbon 2.0, and Prometheus formats. NXLog can exist configured to send data in any of these formats, either by reading preformatted records from a file, or by using an Exec block to output the data in the desired format.
Example 662. Sending host metrics over HTTPS
This configuration illustrates how NXLog can think Microsoft Windows performance counters using the im_winperfcount module and send the information to Sumo Logic in the Carbon ii.0 format using the om_http module.
The input instance is configured to poll the host's available retentiveness every 30 seconds. The field containing the retrieved value is renamed to MemFreeBytes to go far easier for referencing in the output instance.
The Exec block in the output instance builds the metric in the required Carbon 2.0 format. Sumo Logic accepts timestamps in seconds or milliseconds, therefore in this example, the NXLog timestamp is converted from microseconds to milliseconds.
The SUMO_URL value needs to be replaced with an actual Sumo Logic HTTP source URL.
Output sample
The post-obit is a sample of the data that will exist sent to Sumo Logic. In this case, the metric is named Mem_Free and represents the available free memory in bytes.
metric=Mem_Free host=Hopper 1008345088 1608040330345 metric=Mem_Free host=Hopper 1017278464 1608040340346 metric=Mem_Free host=Hopper 1056428032 1608040350346 | Annotation | It may take a few minutes for data to appear in the Sumo Logic web interface. If not, come across the Sumo Logic documentation on troubleshooting HTTP Sources. |
126.6. Verifying data in Sumo Logic
Reception of log data can be verified using the Sumo Logic spider web interface. Ane way to do this is to search for events using the name of the collector and source. Navigate to Manage Data > Collection, and on the Collection tab click on the Open in Log Search icon adjacent to the corresponding log source.
This image displays the NXLog hosted collector and a list of sources in the Sumo Logic web interface. The icon to open up the log search is highlighted.
This image displays events filtered by the collector and source.
This image displays a syslog consequence in JSON format that was sent to Sumo Logic via HTTPS.
This image displays a Windows Event Log issue in JSON format that was sent to Sumo Logic as a syslog message.
Events can exist exported from Sumo Logic in CSV format, in which event fields are output as comma-separated values. The post-obit is a sample syslog message exported from Sumo Logic. The original event was sent by NXLog via HTTPS.
"_messagetimems","_messagetime","_raw","_collector","_size","_source","_sourcecategory","_sourcehost","_sourcename" "1607341156001","12/07/2020 12:39:16.001 +0100","{""EventReceivedTime"":""2020-12-07T12:39:16.001994+01:00"",""SourceModuleName"":""file"",""SourceModuleType"":""im_file"",""SyslogFacilityValue"":i,""SyslogFacility"":""USER"",""SyslogSeverityValue"":5,""SyslogSeverity"":""Detect"",""SeverityValue"":two,""Severity"":""INFO"",""Hostname"":""NXLog-Ubuntu-1"",""EventTime"":""2020-12-07T12:30:37.000000+01:00"",""SourceName"":""systemd"",""ProcessID"":""1"",""Message"":""Started Fingerprint Authentication Daemon.""}","NXLog","414","NXLog-Ubuntu1","linux-http","19.168.0.100","Http Input" Host Metrics tin exist viewed in the Sumo Logic spider web interface past clicking on the + New push to open a Metrics tab, and adding a metric query for the desired property. The image beneath shows host metrics sent by NXLog for available memory (Bytes) over a catamenia of fourth dimension. When sending the information, the metric was named Mem_Free.
Source: https://nxlog.co/documentation/nxlog-user-guide/sumo-logic.html
0 Response to "Sumo Logic to Read Log Data Using Python"
Post a Comment