• Tabbles.net       
  • Chat |
  • Blog |
  • Wiki |
  • Move files with Tabbles instead of Explorer

    F#, C#, WPF and more.
    This is the place to discuss the technology behind Tabbles.

    Re: Move files with Tabbles instead of Explorer

    Postby Leandro » Fri Jul 23, 2010 6:34 pm

    First of all, sorry this bad english... :oops:
    Now, lets get to my report:

    EasyHook is really really easy! 8-)

    I copied the code that I found in EasyHook's manual and started from there. Luckily, the code was just about a "demo application, which logs all file accesses from a given process". That's not what we need, since we're going to keep track of files changes, and not files accesses. But it's very similar and helped me figure out everything.

    That code produces 2 files: an executable that injects a DLL into a given process (like explorer.exe or any other chosen by command-line) and our DLL to be injected, wich intercepts pre-defined calls from the attached process. The executable used for injection continues running, acting like a "host": receiving the DLL log and printing it to the console. The injection terminates as soon as that executable ends.

    The first and crazy thing I did was finding a way to "eliminate" this executable and make it run from a single file, as DLL, if possible... :shock:
    That was the most difficult part, and I did! Why? Because otherwise it would be all the time like this: Tabbles calls DLL Injection executable, that calls EasyHook executable, that calls Injected DLL, that calls LOG treatment's DLL. Everyone running at same time, just to keep track of files. D=
    And I heard that Tabbles already uses 4 or 5 threads together and is becaming memory consuming...

    So, I made a kind of "auto-injectable" DLL! :?:
    I became surprised when it worked, because, at the beginning, everything seemed to be like one DLL for each function to inject, and maybe one executable for each one too. But I managed to do everything in a single call, from a single DLL file! Including all the other functions too! :P
    Oh, I also managed to inject the same DLL in more than one process, from this same file! \o/

    But, of course, the final DLL needs an external little trigger to run:
    Just add a reference to it in your .NET project and call its static "Start" method. The DLL will start, inject itself to the chosen processes, carrying all of our intercepting functions, and stay injected as long as your .NET project executes.


    After injected, our function starts to be called right before the original API. Then we can do whatever we want with its parameters, like logging and updating Tabbles with new path/name of files. Since we need to know if the original function will be suceed or not, to avoid updating Tabbles with fictional changes, I tried to call the original one first and then include its result into the LOG too. But the original one executes without returning to my function, and then, avoiding the entire LOG. I've managed to "fix" this, except by the fact that the original function throws a weird "file not found" error. Weird because everything else keeps working properly, finding, moving files, and logging after this! So, for security purposes, I returned the original function back to the end again. This way, we'll have to store the LOG and check it later to know if the files operations were succeed or not - in other words, if we must update Tabbles or not.

    Once in our functions, we can pass the LOG to other softwares by writing to a log file, or by calling their 3rd-party DLLs, or writing to registry, or by inter-process communications. What's the best way for you? Any other ideas?
    But maybe we should consider performance costs too...

    By writing a log to disk or registry, we should somehow delete it every once in a while, watching performance and data losses.

    By having my injection call a 3rd-party DLL, I'll need to "register" each 3rd-party DLL into a ".INI" file, for example, and call them in a FOR loop, wrapped inside by a try/catch block, to avoid system messes.
    In this case, you'll need to develop your own DLL, install/register it in my ".INI", and then receive my real-time LOG through it. Once received, you'll have your DLL check the result of files operations, and then update Tabbles.

    I also can store this LOG in a static Arraylist inside the DLL too, wich could have its operations results verified after some time, and then accessed, and cleaned, and managed by ... by... oh, I don't know... xD
    But it's not a good idea, because if it comes to happen some crashing, this in-memory LOG vanishes...

    Another important issue is that, using EasyHook, if an already injected process ends, we need to detect its termination and reinject it as soons as it executes again. Otherwise, we might lost track of some files between injections, in case of that process moves or renames them.


    Ok. For now, I guess these are the topics we need to think:
    - Decide what processes we're going to track; (we could even watch and inject every active process if needed!)
    - Decide how to assure a "24-hour injection"; (at windows startup, managed by some "tray-icon-like" program?)
    - Decide how to pass the log for softwares; (I think Tabbles shouldn't be the unique one, for performance purposes)
    - Decide if EasyHook provides our needs; (I'm kind of lazy to analyze alternatives now... :roll: )
    - Anything else?

    As you can see, I guess we need some major planning or brainstorming before continue improving this DLL.
    Since it was practically a two-day-ctrl-C work, I don't care about passing all code to you. :mrgreen:
    This dll is very useful for me too, so feel free to discard it and take another way... ;)

    I'm gonna unwind a bit.
    See'ya later! o/
    Leandro Sá Barreto
    User avatar
    Leandro
    Tabble monkey
    Tabble monkey
     
    Posts: 19
    Joined: Wed Jul 21, 2010 2:28 pm
    Location: Rio de Janeiro, Brazil

    Re: Move files with Tabbles instead of Explorer

    Postby Maurizio » Fri Jul 23, 2010 11:21 pm

    Leandro,

    You rock. I can't believe you did all this work by yourself. I had given up doing this, because it seemed too difficult. And you say this was easy. Really, you are scary. :)

    Now to the technical part. I have read your post and I think I have understood most of it. However, I am not sure your current approach really solves the problem. Let me explain.

    What kind of file accesses does this detect? ______ I assume your dll can detect when a file has been created, deleted, or changed (like .NET's FileSystemWatcher). But not moved. Because there is no "moved" event at the filesystem level.

    If this is true, then it is not easy to detect when a file has been moved. We'll have to deduce that from a sequence of events of the kind created-deleted or deleted-created, close to each other temporally, and with the same file name. Not easy and not reliable.

    For this reason, my idea was to use a different approach and detect calls to MoveFileEx (http://msdn.microsoft.com/en-us/library ... 85%29.aspx) in explorer.exe. So we would still use dll injection, but this time in order to detect Win32 api call, not a filesystem event. What do you think? Have I misunderstood? Could you give an example of what's in the log, so I can better understand if there's enough information to conclude with certainty that a file has been moved?

    Also, in order to use your technique, does the main application need to have administrator privileges? (i.e. elevation)
    User avatar
    Maurizio
    Tabble baker
    Tabble baker
     
    Posts: 196
    Joined: Fri Mar 20, 2009 3:53 pm
    Location: Pescara, Italy

    Re: Move files with Tabbles instead of Explorer

    Postby Leandro » Sat Jul 24, 2010 3:05 am

    Thanks, Maurizio. :)
    Actually, EasyHook owns all of the credits...

    Hum, you really misunderstood me a bit...
    But it's my fault, since I have cut many parts of text and this relevant information became implicit in my bad english... Sorry about that.
    Here's the implicit parts:

    Leandro wrote:"...logs all file accesses from a given process". That's not what we need, since we're going to keep track of files CHANGES, and NOT files accesses.

    Leandro wrote:...like logging and updating Tabbles with NEW path/name of files.

    Leandro wrote:...everything else keeps working properly, finding, MOVING files, and logging after this!

    Leandro wrote:Otherwise, we might lost track of some files between injections, in case of that process MOVES or renames them.



    I didn't mention, to reduce text, but, in fact, I have tested just renaming and moving operations, through windows explorer (right click or drag and drop) and through "openfile dialog" from notepad.exe (with explorer.exe finalized).
    File access were catched too, through a different function, at same time, but I know the difference between them.
    In all of these cases, the console displayed the "LOG" perfectly, as soons as I did the operations!
    But I personalized the LOG's output for cases of moving and renaming files. (wich are the same function though)
    You're gonna see it below.


    Maurizio wrote:What kind of file accesses does this detect?


    Everyone and whatever you want, together or not.

    We're both right because we're both thinking about Win 32 API.
    You just misunderstood me.
    I have injected the simpler version of the function you proposed:
    The MoveFile function from Win 32 API (with just 2 parameters)
    http://msdn.microsoft.com/en-us/library/aa365239(VS.85).aspx
    (We can intercept all of the versions together...)

    That was my injected method for it:

    Code: Select all
    static bool MoveFile_Hooked(
                String InExistingFileName,
                String InNewFileName)
            {
                try
                {
                    Injection This = (Injection)HookRuntimeInfo.Callback;
                    lock (This.Queue)
                    {
                        This.Queue.Push(InExistingFileName + " [MOVED TO] " + InNewFileName + " [RESULT=?]");
                    }
                }
                catch
                {
                }
                return MoveFile(InExistingFileName, InNewFileName);
            }


    This line "could be" the "LOG writing":
    This.Queue.Push("[MOVED FROM] " + InExistingFileName + " [TO] " + InNewFileName + " [RESULT=?]");


    This.Queue.Push() will print this "log" to console window.
    At this time it's just a method call taking a single string as parameter, wich combines a simple personalized output and the received filenames.
    I know it's not what we need, but we can just replace this "print" with anything we want, like passing the filenames as parameters to a 3rd-party DLL, or writing them to a file.

    InExistingFileName is the complete path and file name of the file to be moved to InNewFileName .
    Both of them are absolute paths, and they come directly from the injected process' call to MoveFile API.
    Oh, let me tell you something:
    The function called to just rename a file is also MoveFile.
    So this method catches renamed files too.

    The [RESULT=?] was a previous failed attempt to call the original function first and then include its result at the end of the log's string.
    I replaced the fail code with an '?' char.

    The call to the original API is directly on return command.


    Maurizio wrote:Also, in order to use your technique, does the main application need to have administrator privileges? (i.e. elevation)


    ooops... That is a really really good question! :o
    Detours needs to inject just at installation, but EasyHook needs to inject ALL the time... :shock:
    That's what EasyHook's manual says:

    Injection will sometimes make the Windows Defender complain. (...)
    Such warnings are immediately followed by information pointing out that Windows Defender has
    prevented a malicious attempt. I think this will vanish if you sign all executable binaries of EasyHook
    with AuthentiCode. Such blocking only occurs when injecting into essential system services.


    Please note that injection will fail in most cases if you don’t have admin privileges! Vista is using the UAC for
    evaluating to admin privileges and so you should read some MSDN articles about how to utilize it.


    What are we going to do? Recriate Detours? :cry:
    Now I'm gonna make some tests, studies, and check that good alternatives' list...
    I knew it was getting excessively easy... :(
    See'ya!
    Leandro Sá Barreto
    User avatar
    Leandro
    Tabble monkey
    Tabble monkey
     
    Posts: 19
    Joined: Wed Jul 21, 2010 2:28 pm
    Location: Rio de Janeiro, Brazil

    Re: Move files with Tabbles instead of Explorer

    Postby Leandro » Sat Jul 24, 2010 4:38 pm

    Hey, speaking of injection techniques, I came up with an idea to gain admin privileges inspired on some hacking techniques! :twisted:

    This morning I realized that probably there is no problem on injecting all the time if we register our trigger executable as a Windows Service, at installation! :idea:

    It might run at System Startup, for all users, and thus, have administrator privileges. 8-)

    We can make a specific Tabbles Service if you have similar needs for it. ;)

    I'm working on a general FileTracking Service now. :geek:
    I'll report at the end of the day. See'ya! :D
    Leandro Sá Barreto
    User avatar
    Leandro
    Tabble monkey
    Tabble monkey
     
    Posts: 19
    Joined: Wed Jul 21, 2010 2:28 pm
    Location: Rio de Janeiro, Brazil

    Re: Move files with Tabbles instead of Explorer

    Postby Andrea » Sat Jul 24, 2010 7:02 pm

    Leandro, you are a Dragon:

    Image


    :mrgreen: :mrgreen: :ugeek: :ugeek:
    User avatar
    Andrea
    Tabble baker
    Tabble baker
     
    Posts: 734
    Joined: Sat Jun 20, 2009 9:07 pm
    Location: Copenhagen, Denmark

    Re: Move files with Tabbles instead of Explorer

    Postby Leandro » Sun Jul 25, 2010 10:15 pm

    GGGGRRRRROOOOOOOOOOOOOAAAAAAAAAARRRRR!!!! hehehehe

    Thanks, Andrea!
    I feel honored! :D

    I didn't came here yesterday because the "end of [my] day" was 2:30 AM...
    That was the time I finished my work...

    "Finished"? Yeah! It's done!! \o/ 8-)

    Or almost done... :roll:

    Well, let me first explain what we got:
    Do you remember that unique DLL I told you?
    I split it up again, on 3 projects inside the same Solution:
    -FileTracking.dll, cointains injection stuffs.
    -svFileTracking.exe, it's a registered service that injects the DLL into processes.
    -FileTrackingSetup, performs an automatic installation of everything, including the service register.

    At this moment we have a complete setup that installs 7 files directly into System Folder (system32):
    svFileTracking.exe, FileTracking.dll, and 5 required EasyHook's files. (But it uses just 2 or 3 of them)
    It also creates an application folder inside Program Files.
    That's the place the Service and DLL write their logs to.

    Everything is clean, organized and fine!

    Today I've improved the service and made some tests...
    I've installed it as admin and restarted windows as limited user.
    Using some "hacking" techniques, I could open Task Manager on logon screen, just before any user logs into System.
    There I could see that svFileTracking.exe was already executing on system! :)
    Then I logged in as limited user and saw svFileTracking writing some log...
    BUT, I noticed some log entries was missing, and they should have been wrote... :|
    (...)

    ** UPDATE ** 26/Jul - 02:20 AM

    I deleted this part of the post because I've already fixed the problem! :P
    And that was very shameful because after everything I just couldn't write a simple log file to disk... :oops:
    After analyzing our 90MB's log (hehehehe... I've already fixed this... :mrgreen: ) I realized that, even though our DLL injects fine, it has its log writing blocked by system because it's injected in a process created by a limited user, wich doesn't have write access to log's folder.
    I fixed this by sending the DLL's log to the main service (svFileTracking.exe) and have it wrote from there. :idea:
    I've already told this before: The executable acts like a hosting, receiving data from DLL through an interface provided by EasyHook.
    This way, our service, wich has admin privileges, can write logs or do another "admin tasks" for the DLLs! 8-)
    I could say now we have admin access to system!
    (or not....)

    Now I'm storing some configuration data inside a .INI file. :idea:
    For example, now the service injects into every instances of the processes told by INI file!
    And by using a timed event I could watch the processes and then manage the injections properly... :ugeek:
    There's a piece of code that stores the PID of sucessful injected processes.
    So it only injects into "pure" process, alowing some more high-level management.
    I guess now I could "easily" inject to all running processes and watch for new ones. :geek:

    So, I guess the core is totally developed!
    :D
    Now I'm going to add more APIs, personalize their file's outputs, and improve the ".INI system", adding some handful "INI configurations".
    This INI system should be very practical and utile! :P

    For Tabbles integration, you should discuss this and tell me what's the best way for it. ;)
    We could replace the file log system with another feature, or even add new features to be configured through INI files.
    Possible 3rd-party DLLs could be registered in this INI, for example.


    We can discuss together if needed.
    I saw the google wave post and liked it!
    There are still lots of work ahead! :shock:
    I wish I could help with this...
    But I'm not sure if I could... :?

    May I ask you a question?
    You said there that in the future you could hire distant employees... :)
    How could this be possible? Informally?

    Well, the main work for FileTracking is done. 8-)
    I guess I need to spend some time learning Google Wave yet.
    And Skype as well... So I should delay to show up there...
    See'ya later! :D
    Leandro Sá Barreto
    User avatar
    Leandro
    Tabble monkey
    Tabble monkey
     
    Posts: 19
    Joined: Wed Jul 21, 2010 2:28 pm
    Location: Rio de Janeiro, Brazil

    Re: Move files with Tabbles instead of Explorer

    Postby Maurizio » Mon Jul 26, 2010 2:24 pm

    @Leandro

    Damn I read your new posts just now, because the notification email did not arrive :x And I was expecting you to write on Google Wave. Oh well. ;)

    Anyway, it seems you did a damn good job! Congratulations. If we are ever able to hire someone, you will be the first. I'll leave Andrea to write about the details.

    Some sparse remarks:

    1.
    So it seems at this point we have a Windows service to install. The burden moves to the installer. Currently I don't know how to modify the Tabbles installer to install a service and have to investigate.

    But you say you also wrote an installer. How did you do it? With Visual Studio?

    2.

    The easiest thing to do could be to have the service write the paths of the files moved to a text log, together with the operation timestamp, and then the main Tabbles executable could poll this log periodically, parse it, and extract the lines it hasn't yet read. Or we could use the filesystemwatcher to detect when the text log has changed; this would avoid the need for polling.

    However, this approach has a few problems: efficiency (you have to parse a lot of lines to find the ones you haven't yet processed); delay (if we use polling as opposed to the FileSystemWatcher); and needs a logic to empty the old lines of the log from time to time.

    I am wondering if there is some other technique (other than writing to a txt file) to send messages from the service to tabbles.exe. _______But this would have a disadvantage: it would only work when Tabbles is running. So maybe it's not worth to try this technique, and start with the log file. (Later maybe the techniques can be combined for efficiency)


    3.
    who should start the service (since surely we can't ask the user to start it manually via command line)? Tabbles.exe, at startup? But it does not have admin privileges... I suppose it should always stay running, and never turned off.

    4.
    Now the service injects into every instances of the processes told by INI file


    How do you specify those processes? you pass the path to the executable?


    5.
    about how to integrate this in Tabbles, I am not sure. If I could look at the source code maybe things would be clearer. Would you like to create an open source project for your tool, or do you prefer to keep the sources closed?

    _____

    Thanks again for the wonderful job. If we could continue this chat on Google Wave, it would be much more comfortable.
    User avatar
    Maurizio
    Tabble baker
    Tabble baker
     
    Posts: 196
    Joined: Fri Mar 20, 2009 3:53 pm
    Location: Pescara, Italy

    Re: Move files with Tabbles instead of Explorer

    Postby Andrea » Mon Jul 26, 2010 3:20 pm

    @Leandro,

    I owe you an email now... :-)

    (in the meanwhile, keep on the excellent work, o dragao! :D :lol: :mrgreen: :ugeek: )
    User avatar
    Andrea
    Tabble baker
    Tabble baker
     
    Posts: 734
    Joined: Sat Jun 20, 2009 9:07 pm
    Location: Copenhagen, Denmark

    Re: Move files with Tabbles instead of Explorer

    Postby Leandro » Thu Jul 29, 2010 1:55 pm

    @Maurizio

    Don't worry! It was sunday! ;)

    Thanks for the congratulations! You did good jobs as well!
    Actually, I bet you could develop FileTracking better than me... You have many projects and I'm still at university...

    Sorry the delay, I've spent lots of time writing all of this... And I had to adapt the text to post on Google Wave too.
    But there's no redundant information here or there! I'll try to post in both places until "migrate" to Google Wave.

    Hum, I was working on a time management software that I planned to finish before my classes start again.
    So this should slow things up for FileTracking...
    But I won't abandonate neither of them. ;)

    Let's get to your questions:


    < 1. >
    Yes, Visual Studio did the Setup for me.
    In case of Windows Services, we just add an Installer class for it, besides Setup.
    This class will automatically register and configure the Service during Setup.



    < 3. >
    Maurizio wrote:who should start the service (...)? Tabbles.exe, at startup? But it does not have admin privileges... I suppose it should always stay running, and never turned off.


    Hey, FileTracking has admin privileges! 8-)
    (check my last post, about limited users)

    And, yes, it always stays running, and never turned off!
    The Service is registered on System during installation as an "automatic" Service.
    Just restart the system and it should be already running on logon screen! :P

    (I assure it because I saw it on taskmgr and services.msc, through CMD.exe in place of sethc.exe - It's a kind of "junior hacking" technique...)



    < 4. >
    Maurizio wrote:How do you specify those processes? you pass the path to the executable?


    No, you write the name processes into FileTracking.INI file, located in AppsFolder!
    Everytime the service starts, it loads configurations from this file! I'm still improving this system.

    Below (IPC) you'll see that it's possible to any software edit this configurations and reload them again without restarting system.



    < 2. >
    Maurizio wrote:(Later maybe the techniques can be combined for efficiency)


    That's it! And that's what I've just did! I've combined 2 different features for log.
    The log file system was already "practically" done, and yesterday (monday?) I started to implement an IPC (Inter-Process Communication) between FileTracking and any client softwares (i.e. like Tabbles).
    Most of it are not implemented yet but the main feature is done:
    FileTracking Service creates an IPC server on local machine to listen for client's requests.
    This way, Tabbles or any other softwares can at anytime retrieve every information needed about newest or previous logs, through IPC or log files!


    Now it's really looking like a service!

    How does it work?
    Check this out on Google Wave:
    https://wave.google.com/wave/waveref/go ... +OsNtHUDjB


    < 5. >
    Maurizio wrote:about how to integrate this in Tabbles, I am not sure. If I could look at the source code maybe things would be clearer. Would you like to create an open source project for your tool, or do you prefer to keep the sources closed?



    That's a great idea!
    Hum... how do we do that?

    Anyway, now you can figure out how to integrate using IPC! (check my post above on Google Wave)
    Does this technique provide a good solution for you? You could just add my functions signatures to Tabbles executable and use it as I mentioned.

    At this point we can see FileTracking as an independent software project, just like Tabbles.
    Since FileTracking provides IPC and log files, Tabbles can use FileTracking by 3 ways, combined or separated:
    - By just analyzing the log files (not recommended);
    - By implementing at code level a class for direct communication through IPC;
    - By developing a plugin system where a specific plugin connects FileTracking via IPC.


    Which of these 3 ways do you prefer?

    About plugins, I've never done it, I guess, but I do have a good starting idea about how to do that.
    Or maybe it's not that good... My thoughts about plugins could confuse you... :?
    I'll wait your reply...
    Bye bye!
    Leandro Sá Barreto
    User avatar
    Leandro
    Tabble monkey
    Tabble monkey
     
    Posts: 19
    Joined: Wed Jul 21, 2010 2:28 pm
    Location: Rio de Janeiro, Brazil

    Re: Move files with Tabbles instead of Explorer

    Postby Maurizio » Thu Jul 29, 2010 2:23 pm

    Hum, I was working on a time management software that I planned to finish before my classes start again.
    So this should slow things up for FileTracking...
    But I won't abandonate neither of them. ;)


    Don't worry, you've already done more than we expected!





    ...

    < 2. >
    Maurizio wrote:(Later maybe the techniques can be combined for efficiency)


    That's it! And that's what I've just did! I've combined 2 different features for log.
    The log file system was already "practically" done, and yesterday (monday?) I started to implement an IPC (Inter-Process Communication) between FileTracking and any client softwares (i.e. like Tabbles).
    Most of it are not implemented yet but the main feature is done:
    FileTracking Service creates an IPC server on local machine to listen for client's requests.
    This way, Tabbles or any other softwares can at anytime retrieve every information needed about newest or previous logs, through IPC or log files!


    awesome!


    Maurizio wrote:about how to integrate this in Tabbles, I am not sure. If I could look at the source code maybe things would be clearer. Would you like to create an open source project for your tool, or do you prefer to keep the sources closed?



    That's a great idea!
    Hum... how do we do that?


    I am not sure. You can think this later, you can release the code when you like, you don't have to do it now. _____Anyway, I think maybe you could create a project on codeplex and put the software there. Then you have to choose one or more licenses (so that users must choose one of those licenses before using your code). If you only offer LGPL, we (Tabbles) will only be able to add your dll to our project, but not entitled to modify your code. If you also choose a more permissive license, we will also be able to modify the code. I don't have a lot of experience with licensing though.


    Anyway, now you can figure out how to integrate using IPC! (check my post above on Google Wave)
    Does this technique provide a good solution for you? You could just add my functions signatures to Tabbles executable and use it as I mentioned.


    So far it seems a perfect solution. But I have yet to understand better how it works.

    At this point we can see FileTracking as an independent software project, just like Tabbles.
    Since FileTracking provides IPC and log files, Tabbles can use FileTracking by 3 ways, combined or separated:
    - By just analyzing the log files (not recommended);
    - By implementing at code level a class for direct communication trough IPC;
    - By developing a plugin system where a specific plugin connects FileTracking via IPC.

    Which of these 3 ways do you prefer?


    For now, surely the first one, to get it running quickly and see if there are problems. The second one we can add later, when we want to optimize things. The third one I don't understand :)

    I am going to wave to comment there. Thank you!
    User avatar
    Maurizio
    Tabble baker
    Tabble baker
     
    Posts: 196
    Joined: Fri Mar 20, 2009 3:53 pm
    Location: Pescara, Italy

    PreviousNext

    • Similar topics
      Replies
      Views
      Author

    Return to Development

    Who is online

    Users browsing this forum: No registered users and 1 guest

    cron