This chapter contains recipes for common tasks that can be done in lnav. These recipes can be used as a starting point for your own needs after some adaptation.
Defining a New Format#
Log messages can be annotated in a couple of different ways in lnav to help you get organized.
Create partitions for Linux boots#
When digging through logs that can be broken up into multiple sections, lnav’s partitioning feature can be used to keep track of which section you are in. For example, if a collection of Linux logs covered multiple boots, the following script could be used to create partitions for each boot. After the partition name is set for the log messages, the current name will show up in the top status bar next to the current time.
1# 2# DO NOT EDIT THIS FILE, IT WILL BE OVERWRITTEN! 3# 4# @synopsis: partition-by-boot 5# @description: Partition the log view based on boot messages from the Linux kernel. 6# 7 8;UPDATE syslog_log 9 SET log_part = 'Boot: ' || log_time 10 WHERE log_text LIKE '%kernel:%Linux version%'; 11 12;SELECT 'Created ' || changes() || ' partitions(s)';
Tagging SSH log messages#
Log messages can be tagged interactively with the :tag command or programmatically using the SQLite Interface. This example uses a script to search for interesting SSH messages and automatically adds an appropriate tag.
1# 2# @synopsis: tag-ssh-msgs 3# @description: Tag interesting SSH log messages 4# 5 6;UPDATE all_logs 7 SET log_tags = json_concat(log_tags, '#ssh.invalid-user') 8 WHERE log_text LIKE '%Invalid user from%' 9 10;SELECT 'Tagged ' || changes() || ' messages';
Most log analysis within lnav is done through the SQLite Interface. The
following examples should give you some ideas to start leveraging this
functionality. One thing to keep in mind is that if a query gets to be too
large or multiple statements need to be executed, you can create a
.lnav script that contains the statements and execute it using the
| command prompt.
Count client IPs in web access logs#
To count the occurrences of an IP in web access logs and order the results from highest to lowest:
;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC
Show only lines where a numeric field is in a range#
The :filter-expr command can be used to filter web access logs to only show lines where the number of bytes transferred to the client is between 10,000 and 40,000 bytes like so:
:filter-expr :sc_bytes BETWEEN 10000 AND 40000
Generating a Report#
Reports can be generated by writing an lnav script that uses SQL queries and commands to format a document. A basic script can simply execute a SQL query that is shown in the DB view. More sophisticated scripts can use the following commands to generate customized output for a report:
The :echo command to write plain text
1# 2# @synopsis: report-demo [<output-path>] 3# @description: Generate a report for requests in access_log files 4# 5 6# Figure out the file path where the report should be written to, default is 7# stdout 8;SELECT CASE 9 WHEN $1 IS NULL THEN '-' 10 ELSE $1 11 END AS out_path 12 13# Redirect output from commands to $out_path 14:redirect-to $out_path 15 16# Print an introductory message 17;SELECT printf('\n%d total requests', count(1)) AS msg FROM access_log 18:echo $msg 19 20;WITH top_paths AS ( 21 SELECT 22 cs_uri_stem, 23 count(1) AS total_hits, 24 sum(sc_bytes) as bytes, 25 count(distinct c_ip) as visitors 26 FROM access_log 27 WHERE sc_status BETWEEN 200 AND 300 28 GROUP BY cs_uri_stem 29 ORDER BY total_hits DESC 30 LIMIT 50), 31 weekly_hits_with_gaps AS ( 32 SELECT timeslice(log_time_msecs, '1w') AS week, 33 cs_uri_stem, 34 count(1) AS weekly_hits 35 FROM access_log 36 WHERE cs_uri_stem IN (SELECT cs_uri_stem FROM top_paths) AND 37 sc_status BETWEEN 200 AND 300 38 GROUP BY week, cs_uri_stem), 39 all_weeks AS ( 40 SELECT week 41 FROM weekly_hits_with_gaps 42 GROUP BY week 43 ORDER BY week ASC), 44 weekly_hits AS ( 45 SELECT all_weeks.week, 46 top_paths.cs_uri_stem, 47 ifnull(weekly_hits, 0) AS hits 48 FROM all_weeks 49 CROSS JOIN top_paths 50 LEFT JOIN weekly_hits_with_gaps 51 ON all_weeks.week = weekly_hits_with_gaps.week AND 52 top_paths.cs_uri_stem = weekly_hits_with_gaps.cs_uri_stem) 53 SELECT weekly_hits.cs_uri_stem AS Path, 54 printf('%,9d', total_hits) AS Hits, 55 printf('%,9d', visitors) AS Visitors, 56 printf('%9s', humanize_file_size(bytes)) as Amount, 57 sparkline(hits) AS Weeks 58 FROM weekly_hits 59 LEFT JOIN top_paths ON top_paths.cs_uri_stem = weekly_hits.cs_uri_stem 60 GROUP BY weekly_hits.cs_uri_stem 61 ORDER BY Hits DESC 62 LIMIT 10 63 64:write-table-to - 65 66:echo 67:echo Failed Requests 68:echo 69 70;SELECT printf('%,9d', count(1)) AS Hits, 71 printf('%,9d', count(distinct c_ip)) AS Visitors, 72 sc_status AS Status, 73 cs_method AS Method, 74 group_concat(distinct cs_version) AS Versions, 75 cs_uri_stem AS Path, 76 replicate('|', (cast(count(1) AS REAL) / $total_requests) * 100.0) AS "% of Requests" 77 FROM access_log 78 WHERE sc_status >= 400 79 GROUP BY cs_method, cs_uri_stem 80 ORDER BY Hits DESC 81 LIMIT 10 82 83:write-table-to -