iwla

iwla Git Source Tree

Root/docs/modules.md

1 * iwla.py
2 * plugins/display/all_visits.py
3 * plugins/display/browsers.py
4 * plugins/display/feeds.py
5 * plugins/display/filter_users.py
6 * plugins/display/hours_stats.py
7 * plugins/display/ip_to_geo.py
8 * plugins/display/istats_diff.py
9 * plugins/display/operating_systems.py
10 * plugins/display/referers_diff.py
11 * plugins/display/referers.py
12 * plugins/display/robot_bandwidth.py
13 * plugins/display/top_downloads_diff.py
14 * plugins/display/top_downloads.py
15 * plugins/display/top_hits.py
16 * plugins/display/top_pages_diff.py
17 * plugins/display/top_pages.py
18 * plugins/display/top_visitors.py
19 * plugins/display/track_users.py
20 * plugins/post_analysis/browsers.py
21 * plugins/post_analysis/feeds.py
22 * plugins/post_analysis/hours_stats.py
23 * plugins/post_analysis/ip_to_geo.py
24 * plugins/post_analysis/iptogeo.py
25 * plugins/post_analysis/iptogeo.reset.py
26 * plugins/post_analysis/operating_systems.py
27 * plugins/post_analysis/referers.py
28 * plugins/post_analysis/reverse_dns.py
29 * plugins/post_analysis/top_downloads.py
30 * plugins/post_analysis/top_hits.py
31 * plugins/post_analysis/top_pages.py
32 * plugins/pre_analysis/page_to_hit.py
33 * plugins/pre_analysis/robots.py
34
35
36iwla
37----
38
39 Main class IWLA
40 Parse Log, compute them, call plugins and produce output
41 For now, only HTTP log are valid
42
43 Plugin requirements :
44 None
45
46 Conf values needed :
47 analyzed_filename
48 domain_name
49 locales_path
50 compress_output_files
51 excluded_ip
52
53 Output files :
54 DB_ROOT/meta.db
55 DB_ROOT/year/month/iwla.db
56 OUTPUT_ROOT/index.html
57 OUTPUT_ROOT/year/_stats.html
58 OUTPUT_ROOT/year/month/index.html
59
60 Statistics creation :
61
62 meta :
63 last_time
64 start_analysis_time
65 stats =>
66 year =>
67 month =>
68 viewed_bandwidth
69 not_viewed_bandwidth
70 viewed_pages
71 viewed_hits
72 nb_visits
73 nb_visitors
74
75 month_stats :
76 viewed_bandwidth
77 not_viewed_bandwidth
78 viewed_pages
79 viewed_hits
80 nb_visits
81
82 days_stats :
83 day =>
84 viewed_bandwidth
85 not_viewed_bandwidth
86 viewed_pages
87 viewed_hits
88 nb_visits
89 nb_visitors
90
91 visits :
92 remote_addr =>
93 remote_addr
94 remote_ip
95 viewed_pages{0..31} # 0 contains total
96 viewed_hits{0..31} # 0 contains total
97 not_viewed_pages{0..31}
98 not_viewed_hits{0..31}
99 bandwidth{0..31}
100 last_access
101 requests =>
102 [fields_from_format_log]
103 extract_request =>
104 http_method
105 http_uri
106 http_version
107 extract_uri
108 extract_parameters*
109 extract_referer* =>
110 extract_uri
111 extract_parameters*
112 robot
113 hit_only
114 is_page
115
116 valid_visitors:
117 month_stats without robot and hit only visitors (if not conf.count_hit_only_visitors)
118
119 Statistics update :
120 None
121
122 Statistics deletion :
123 None
124
125
126plugins.display.all_visits
127--------------------------
128
129 Display hook
130
131 Create All visits page
132
133 Plugin requirements :
134 None
135
136 Conf values needed :
137 display_visitor_ip*
138
139 Output files :
140 OUTPUT_ROOT/year/month/all_visits.html
141 OUTPUT_ROOT/year/month/index.html
142
143 Statistics creation :
144 None
145
146 Statistics update :
147 None
148
149 Statistics deletion :
150 None
151
152
153plugins.display.browsers
154------------------------
155
156 Display hook
157
158 Create browsers page
159
160 Plugin requirements :
161 post_analysis/browsers
162
163 Conf values needed :
164 max_browsers_displayed*
165 create_browsers_page*
166
167 Output files :
168 OUTPUT_ROOT/year/month/browsers.html
169 OUTPUT_ROOT/year/month/index.html
170
171 Statistics creation :
172 None
173
174 Statistics update :
175 None
176
177 Statistics deletion :
178 None
179
180
181plugins.display.feeds
182---------------------
183
184 Display hook
185
186 Display feeds parsers
187
188 Plugin requirements :
189 post_analysis/feeds
190
191 Conf values needed :
192 create_all_feeds_page*
193
194 Output files :
195 OUTPUT_ROOT/year/month/index.html
196 OUTPUT_ROOT/year/month/all_feeds.html
197
198 Statistics creation :
199 None
200
201 Statistics update :
202 None
203
204 Statistics deletion :
205 None
206
207
208plugins.display.filter_users
209----------------------------
210
211 Display hook
212
213 Filter users
214
215 Plugin requirements :
216 None
217
218 Conf values needed :
219 filtered_users : list of filters
220 filtered_ip : list of ip (string)
221 create_filtered_page*
222
223 Filter is a list of filter description combined by AND operator
224 Filter description is a list of 3 elements :
225
226 * Field to match in visits
227 * Operator '=', '==', '!=', '>', '>=', '<', '<=' for int value
228 * Operator '=', '==', '!=', 'in', 'match' for str value
229 * Target value
230
231 For easiest config, you can indicate both 'remote_addr' or 'ip' in field element
232
233 Output files :
234 OUTPUT_ROOT/year/month/index.html
235 OUTPUT_ROOT/year/month/filtered_users.html
236
237 Statistics creation :
238 None
239
240 Statistics update :
241 None
242
243 Statistics deletion :
244 None
245
246
247plugins.display.hours_stats
248---------------------------
249
250 Display hook
251
252 Display statistics by hour/week day
253
254 Plugin requirements :
255 post_analysis/hours_stats
256
257 Conf values needed :
258 None
259
260 Output files :
261 OUTPUT_ROOT/year/month/index.html
262
263 Statistics creation :
264 None
265
266 Statistics update :
267 None
268
269 Statistics deletion :
270 None
271
272
273plugins.display.ip_to_geo
274-------------------------
275
276 Display hook
277
278 Add geo statistics
279
280 Plugin requirements :
281 post_analysis/ip_to_geo
282
283 Conf values needed :
284 create_geo_page*
285
286 Output files :
287 OUTPUT_ROOT/year/month/index.html
288
289 Statistics creation :
290 None
291
292 Statistics update :
293 None
294
295 Statistics deletion :
296 None
297
298
299plugins.display.istats_diff
300---------------------------
301
302 Display hook interface
303
304 Enlight new and updated statistics
305
306 Plugin requirements :
307 None
308
309 Conf values needed :
310 None
311
312 Output files :
313 None
314
315 Statistics creation :
316 None
317
318 Statistics update :
319 None
320
321 Statistics deletion :
322 None
323
324
325plugins.display.operating_systems
326---------------------------------
327
328 Display hook
329
330 Add operating systems statistics
331
332 Plugin requirements :
333 post_analysis/operating_systems
334
335 Conf values needed :
336 create_families_page*
337
338 Output files :
339 OUTPUT_ROOT/year/month/index.html
340
341 Statistics creation :
342 None
343
344 Statistics update :
345 None
346
347 Statistics deletion :
348 None
349
350
351plugins.display.referers_diff
352-----------------------------
353
354 Display hook
355
356 Enlight new and updated key phrases in in all_key_phrases.html
357
358 Plugin requirements :
359 display/referers
360
361 Conf values needed :
362 None
363
364 Output files :
365 None
366
367 Statistics creation :
368 None
369
370 Statistics update :
371 None
372
373 Statistics deletion :
374 None
375
376
377plugins.display.referers
378------------------------
379
380 Display hook
381
382 Create Referers page
383
384 Plugin requirements :
385 post_analysis/referers
386
387 Conf values needed :
388 max_referers_displayed*
389 create_all_referers_page*
390 max_key_phrases_displayed*
391 create_all_key_phrases_page*
392
393 Output files :
394 OUTPUT_ROOT/year/month/referers.html
395 OUTPUT_ROOT/year/month/key_phrases.html
396 OUTPUT_ROOT/year/month/index.html
397
398 Statistics creation :
399 None
400
401 Statistics update :
402 None
403
404 Statistics deletion :
405 None
406
407
408plugins.display.robot_bandwidth
409-------------------------------
410
411 Display hook
412
413 Display top 10 robot bandwidth use
414
415 Plugin requirements :
416 None
417
418 Conf values needed :
419 display_visitor_ip*
420 create_all_robot_bandwidth_page*
421
422 Output files :
423 OUTPUT_ROOT/year/month/top_robots_bandwidth.html
424 OUTPUT_ROOT/year/month/index.html
425
426 Statistics creation :
427 None
428
429 Statistics update :
430 None
431
432 Statistics deletion :
433 None
434
435
436plugins.display.top_downloads_diff
437----------------------------------
438
439 Display hook
440
441 Enlight new and updated downloads in in top_downloads.html
442
443 Plugin requirements :
444 display/top_downloads
445
446 Conf values needed :
447 None
448
449 Output files :
450 None
451
452 Statistics creation :
453 None
454
455 Statistics update :
456 None
457
458 Statistics deletion :
459 None
460
461
462plugins.display.top_downloads
463-----------------------------
464
465 Display hook
466
467 Create TOP downloads page
468
469 Plugin requirements :
470 post_analysis/top_downloads
471
472 Conf values needed :
473 max_downloads_displayed*
474 create_all_downloads_page*
475
476 Output files :
477 OUTPUT_ROOT/year/month/top_downloads.html
478 OUTPUT_ROOT/year/month/index.html
479
480 Statistics creation :
481 None
482
483 Statistics update :
484 None
485
486 Statistics deletion :
487 None
488
489
490plugins.display.top_hits
491------------------------
492
493 Display hook
494
495 Create TOP hits page
496
497 Plugin requirements :
498 post_analysis/top_hits
499
500 Conf values needed :
501 max_hits_displayed*
502 create_all_hits_page*
503
504 Output files :
505 OUTPUT_ROOT/year/month/top_hits.html
506 OUTPUT_ROOT/year/month/index.html
507
508 Statistics creation :
509 None
510
511 Statistics update :
512 None
513
514 Statistics deletion :
515 None
516
517
518plugins.display.top_pages_diff
519------------------------------
520
521 Display hook
522
523 Enlight new and updated pages in in top_pages.html
524
525 Plugin requirements :
526 display/top_pages
527
528 Conf values needed :
529 None
530
531 Output files :
532 None
533
534 Statistics creation :
535 None
536
537 Statistics update :
538 None
539
540 Statistics deletion :
541 None
542
543
544plugins.display.top_pages
545-------------------------
546
547 Display hook
548
549 Create TOP pages page
550
551 Plugin requirements :
552 post_analysis/top_pages
553
554 Conf values needed :
555 max_pages_displayed*
556 create_all_pages_page*
557
558 Output files :
559 OUTPUT_ROOT/year/month/top_pages.html
560 OUTPUT_ROOT/year/month/index.html
561
562 Statistics creation :
563 None
564
565 Statistics update :
566 None
567
568 Statistics deletion :
569 None
570
571
572plugins.display.top_visitors
573----------------------------
574
575 Display hook
576
577 Create TOP visitors block
578
579 Plugin requirements :
580 None
581
582 Conf values needed :
583 display_visitor_ip*
584
585 Output files :
586 OUTPUT_ROOT/year/month/index.html
587
588 Statistics creation :
589 None
590
591 Statistics update :
592 None
593
594 Statistics deletion :
595 None
596
597
598plugins.display.track_users
599---------------------------
600
601 Display hook
602
603 Track users
604
605 Plugin requirements :
606 None
607
608 Conf values needed :
609 tracked_ip
610 create_tracked_page*
611
612 Output files :
613 OUTPUT_ROOT/year/month/index.html
614 OUTPUT_ROOT/year/month/tracked_users.html
615
616 Statistics creation :
617 None
618
619 Statistics update :
620 None
621
622 Statistics deletion :
623 None
624
625
626plugins.post_analysis.browsers
627------------------------------
628
629 Post analysis hook
630
631 Detect browser information from requests
632
633 Plugin requirements :
634 None
635
636 Conf values needed :
637 None
638
639 Output files :
640 None
641
642 Statistics creation :
643 visits :
644 remote_addr =>
645 browser
646
647 month_stats :
648 browsers =>
649 browser => count
650
651 Statistics update :
652 None
653
654 Statistics deletion :
655 None
656
657
658plugins.post_analysis.feeds
659---------------------------
660
661 Post analysis hook
662
663 Find feeds parsers (first hit in feeds conf value and no viewed pages if it's a robot)
664 If there is ony one hit per day to a feed, merge feeds parsers with the same user agent
665 as it must be the same person with a different IP address.
666
667 Plugin requirements :
668 None
669
670 Conf values needed :
671 feeds
672 feeds_referers*
673 merge_one_hit_only_feeds_parsers*
674
675 Output files :
676 None
677
678 Statistics creation :
679 remote_addr =>
680 feed_parser
681
682 Statistics update :
683 None
684
685 Statistics deletion :
686 None
687
688
689plugins.post_analysis.hours_stats
690---------------------------------
691
692 Post analysis hook
693
694 Count pages, hits and bandwidth by hour/week day
695
696 Plugin requirements :
697 None
698
699 Conf values needed :
700 None
701
702 Output files :
703 None
704
705 Statistics creation :
706 month_stats:
707 hours_stats =>
708 00 .. 23 =>
709 pages
710 hits
711 bandwidth
712
713 days_stats =>
714 0 .. 6 =>
715 pages
716 hits
717 bandwidth
718
719 Statistics update :
720 None
721
722 Statistics deletion :
723 None
724
725
726plugins.post_analysis.ip_to_geo
727-------------------------------
728
729 Post analysis hook
730
731 Get country code from IP address
732
733 Plugin requirements :
734 None
735
736 Conf values needed :
737 iptogeo_remote_addr*
738 iptogeo_remote_port*
739
740 Output files :
741 None
742
743 Statistics creation :
744 geo =>
745 country_code => count
746 None
747
748 Statistics update :
749 valid_visitors:
750 country_code
751
752 Statistics deletion :
753 None
754
755
756plugins.post_analysis.iptogeo
757-----------------------------
758
759
760
761plugins.post_analysis.iptogeo.reset
762-----------------------------------
763
764
765
766plugins.post_analysis.operating_systems
767---------------------------------------
768
769 Post analysis hook
770
771 Detect operating systems from requests
772
773 Plugin requirements :
774 None
775
776 Conf values needed :
777 None
778
779 Output files :
780 None
781
782 Statistics creation :
783 visits :
784 remote_addr =>
785 operating_system
786
787 month_stats :
788 operating_systems =>
789 operating_system => count
790
791 os_families =>
792 family => count
793
794 Statistics update :
795 None
796
797 Statistics deletion :
798 None
799
800
801plugins.post_analysis.referers
802------------------------------
803
804 Post analysis hook
805
806 Extract referers and key phrases from requests
807
808 Plugin requirements :
809 None
810
811 Conf values needed :
812 domain_name
813
814 Output files :
815 None
816
817 Statistics creation :
818 None
819
820 Statistics update :
821 month_stats :
822 referers =>
823 pages => count
824 hits => count
825 robots_referers =>
826 pages => count
827 hits => count
828 search_engine_referers =>
829 pages => count
830 hits => count
831 key_phrases =>
832 phrase => count
833
834 Statistics deletion :
835 None
836
837
838plugins.post_analysis.reverse_dns
839---------------------------------
840
841 Post analysis hook
842
843 Replace IP by reverse DNS names
844
845 Plugin requirements :
846 None
847
848 Conf values needed :
849 reverse_dns_timeout*
850
851 Output files :
852 None
853
854 Statistics creation :
855 None
856
857 Statistics update :
858 valid_visitors:
859 remote_addr
860 dns_name_replaced
861 dns_analyzed
862
863 Statistics deletion :
864 None
865
866
867plugins.post_analysis.top_downloads
868-----------------------------------
869
870 Post analysis hook
871
872 Count TOP downloads
873
874 Plugin requirements :
875 None
876
877 Conf values needed :
878 None
879
880 Output files :
881 None
882
883 Statistics creation :
884 None
885
886 Statistics update :
887 month_stats:
888 top_downloads =>
889 uri => count
890
891 Statistics deletion :
892 None
893
894
895plugins.post_analysis.top_hits
896------------------------------
897
898 Post analysis hook
899
900 Count TOP hits
901
902 Plugin requirements :
903 None
904
905 Conf values needed :
906 None
907
908 Output files :
909 None
910
911 Statistics creation :
912 None
913
914 Statistics update :
915 month_stats:
916 top_hits =>
917 uri => count
918
919 Statistics deletion :
920 None
921
922
923plugins.post_analysis.top_pages
924-------------------------------
925
926 Post analysis hook
927
928 Count TOP pages
929
930 Plugin requirements :
931 None
932
933 Conf values needed :
934 None
935
936 Output files :
937 None
938
939 Statistics creation :
940 None
941
942 Statistics update :
943 month_stats:
944 top_pages =>
945 uri => count
946
947 Statistics deletion :
948 None
949
950
951plugins.pre_analysis.page_to_hit
952--------------------------------
953
954 Pre analysis hook
955 Change page into hit and hit into page into statistics
956
957 Plugin requirements :
958 None
959
960 Conf values needed :
961 page_to_hit_conf*
962 hit_to_page_conf*
963
964 Output files :
965 None
966
967 Statistics creation :
968 None
969
970 Statistics update :
971 visits :
972 remote_addr =>
973 is_page
974
975 Statistics deletion :
976 None
977
978
979plugins.pre_analysis.robots
980---------------------------
981
982 Pre analysis hook
983
984 Filter robots
985
986 Plugin requirements :
987 None
988
989 Conf values needed :
990 page_to_hit_conf*
991 hit_to_page_conf*
992
993 Output files :
994 None
995
996 Statistics creation :
997 None
998
999 Statistics update :
1000 visits :
1001 remote_addr =>
1002 robot
1003
1004 Statistics deletion :
1005 None
1006
1007

Archive Download this file

Branches

Tags