• 欢迎访问本站网站,推荐使用最新版火狐浏览器和Chrome浏览器访问本网站,如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

在Eclipse下使用代码对网站用户访问的路径日志进行分析

大数据技术与原理 admin 3个月前 (10-18) 63次浏览 已收录 0个评论 扫描二维码
首先确保Eclipse能成功连接到HDSF文件系统
在Eclipse下使用代码对网站用户访问的路径日志进行分析
使用Map方法获取数据
在Eclipse下使用代码对网站用户访问的路径日志进行分析
使用Reduce方法进行统计
在Eclipse下使用代码对网站用户访问的路径日志进行分析
Job提交工作
在Eclipse下使用代码对网站用户访问的路径日志进行分析

 

运行

在Eclipse下使用代码对网站用户访问的路径日志进行分析

运行结果

19/11/22 21:44:51 INFO mapred.Task:  Using ResourceCalculatorProcessTree : null
19/11/22 21:44:51 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@69b4454d
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=1310195712, maxSingleShuffleLimit=327548928, mergeThreshold=864729216, ioSortFactor=10, memToMemMergeOutputsThreshold=10
19/11/22 21:44:51 INFO reduce.EventFetcher: attempt_local1789678123_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
19/11/22 21:44:51 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1789678123_0001_m_000000_0 decomp: 47137112 len: 47137116 to MEMORY
19/11/22 21:44:51 INFO reduce.InMemoryMapOutput: Read 47137112 bytes from map-output for attempt_local1789678123_0001_m_000000_0
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 47137112, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->47137112
19/11/22 21:44:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
19/11/22 21:44:51 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
19/11/22 21:44:51 INFO mapred.Merger: Merging 1 sorted segments
19/11/22 21:44:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 47136939 bytes
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: Merged 1 segments, 47137112 bytes to disk to satisfy reduce memory limit
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: Merging 1 files, 47137116 bytes from disk
19/11/22 21:44:51 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
19/11/22 21:44:51 INFO mapred.Merger: Merging 1 sorted segments
19/11/22 21:44:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 47136939 bytes
19/11/22 21:44:51 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/11/22 21:44:51 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
19/11/22 21:44:52 INFO mapreduce.Job:  map 100% reduce 0%
19/11/22 21:44:53 INFO mapred.Task: Task:attempt_local1789678123_0001_r_000000_0 is done. And is in the process of committing
19/11/22 21:44:53 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/11/22 21:44:53 INFO mapred.Task: Task attempt_local1789678123_0001_r_000000_0 is allowed to commit now
19/11/22 21:44:53 INFO output.FileOutputCommitter: Saved output of task ‘attempt_local1789678123_0001_r_000000_0’ to hdfs://100.64.46.80:8020/output/_temporary/0/task_local1789678123_0001_r_000000
19/11/22 21:44:53 INFO mapred.LocalJobRunner: reduce > reduce
19/11/22 21:44:53 INFO mapred.Task: Task ‘attempt_local1789678123_0001_r_000000_0’ done.
19/11/22 21:44:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local1789678123_0001_r_000000_0
19/11/22 21:44:53 INFO mapred.LocalJobRunner: reduce task executor complete.
19/11/22 21:44:54 INFO mapreduce.Job:  map 100% reduce 100%
19/11/22 21:44:54 INFO mapreduce.Job: Job job_local1789678123_0001 completed successfully
19/11/22 21:44:54 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=94274600
FILE: Number of bytes written=141965372
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=206138930
HDFS: Number of bytes written=41177727
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Map-Reduce Framework
Map input records=381686
Map output records=381686
Map output bytes=46237878
Map output materialized bytes=47137116
Input split bytes=113
Combine input records=0
Combine output records=0
Reduce input groups=7363
Reduce shuffle bytes=47137116
Reduce input records=381686
Reduce output records=7363
Spilled Records=763372
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=36
Total committed heap usage (bytes)=1173356544
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=103069465
File Output Format Counters
Bytes Written=41177727
0

运行成功之后会自动创建一个output目录用来存放分析之后的文件

 

在Eclipse下使用代码对网站用户访问的路径日志进行分析

代码下载:链接:https://pan.baidu.com/s/1627tSskP7M0oR1NXI0Hyjg
提取码:hfoa
解压密码:http://www.ddoslinux.com

本站的文章和资源来自互联网或者站长的原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:在Eclipse下使用代码对网站用户访问的路径日志进行分析
喜欢 (0)
[]
分享 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址